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Foreword 


Iam happy to write this foreword for Beginner’s Guide for Systematic Reviews. 
This book is mainly aimed at public health and social science researchers for 
undertaking systematic reviews. The aim of this guide is to promote high 
standards in commissioning, conducting, and providing practical guidance for 
undertaking systematic reviews evaluating the effects of health interventions. 


Over the last decade, the demand for use of the best available research 
evidence to inform health care decision making and public policy has increased 
considerably. Systematic reviews aim to identify, appraise, and summarize 
the findings from all relevant individual studies on a topic. Well conducted 
Systematic Reviews provide the best evidence to guide clinical practice, they are 
considered a cornerstone for the recommendations of Evidence-based practice 
guidelines and should be an integral part of planning future research activities. 
The Beginner's Guide has been written for those with an understanding of health 
research & seeking skills for conducting systematic reviews. The guide is also 
aimed at those who commission systematic reviews, such as the government and 
funding organisations. Though the book is mainly aimed at health practitioners, 
some researchers working on systematic reviews on social sciences might also 
find it useful. 


I appreciate the efforts of the contributors in preparing the Beginner’s Guide. As 
we release this guide, I envision that it would serve a variety of audiences such 
as policymakers, researchers, clinicians and other health professionals who 
need to be aware of evidence-informed decision-making in health care. 


Balfron Broygouw 


re 


(Dr. Balram Bhargava) 


Tele.: 26588204, 26589620, Fax (Off.) : 91-1 1-26588662, E-mail: secy-dg@icmr.gov.in 
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1. Introduction 


1.1 What Is a Systematic Review? 


A systematic review is a review of the evidence on a clearly formulated 
question that uses systematic and explicit methods to identify, select and 
critically appraise relevant primary research, and to extract and analyze 
data from the studies that are included in the reviews'. 


Diverse range of information sources and an explosion of knowledge have 
made it impossible for clinical researchers to stay abreast with advances in a 
given field. Systematic reviews and meta-analyses were conducted in health 
in 1970s and 1980s. Both the terms meta-analyses and systematic reviews 
were used interchangeably and often created confusion among the readers. 
Chalmers and Altman’ suggested that the term ‘Meta-Analysis’ may be 
used only for statistical synthesis or quantitative methods of combining 
the evidence from individual studies and it may or may not be part of a 
systematic review. 


Systematic reviews are most often used to study the effectiveness of a 
particular drug treatment compared with a placebo or any other alternative 
treatment. However, these cover a wide range of other issues like: 


e Surgical and nursing techniques e.g. the best ways of carrying out knee 
replacements or the best methods of dressing for chronic wounds 


e Psychosocial interventions e.g. community-based interventions for 
people with schizophrenia 


e Public-health interventions e.g. impact of mobile health (mHealth) 
interventions in health care delivery or lifestyle interventions in 
reducing the prevalence of type II diabetes 


e Adverse effects of drugs or other treatments 


e Economic evaluations e.g. evaluation of implementation intervention 
in public health or drug trials to identify which intervention is cost 
effective 


e Although Systematic Reviews are predominantly used in intervention 
studies, researchers also use this technique in economic evaluation, 
diagnostics tests accuracy, prevalence/incidence etc. 


1.2 Why do we need a systematic review? 
Systematic reviews are needed for the following reasons 


a. To keep abreast of all previous and new research 
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b. To introduce a new treatment that is expected to be better than an 
existing one 


c. To discontinue an old treatment which might be out dated, harmful or 
less cost effective 


d. To draft guidelines for health and social interventions or treatment 
management 


e. To arrive at a consensus where conflicting evidence is reported 


1.3 What is the difference between a narrative review and a 
systematic review? 


In comparison with literature or traditional narrative reviews, systematic 
reviews are much time-intensive and need a research team with multiple 
skills and contributions. Table 1 provides a description of these differences. 


There are some cases where systematic reviews are unable to meet the 
necessary objectives of the review question. In such a case, scoping reviews 
(which are sometimes called scoping exercises/scoping studies) may be more 
useful to consider. Table 2 provides the characteristics of narrative reviews, 
scoping reviews and systematic reviews. 


Table 1: Differences between a narrative review and a systematic review 


| Narrative Review Systematic Review 
Goals Provides summary or overview | Answers a focussed review 
of topic question 
Question Can have a broad topic or a Clearly defined review 
specific question. Hypothesis question using PICO as a 
might not be stated. guide. Hypothesis is stated. 
Authors One or more Three or more 
Protocol No protocol A peer review protocol or 
plan is included 
Objectives May or may not be identified Has clearly stated 
objectives 
Inclusion/ Criteria not usually specified Criteria stated before the 
exclusion criteria review is conducted 


Search strategy | No detailed search strategy, Detailed and 
mostly conducted using comprehensive search 
keywords and snow-balling strategy 


Introduction 


Narrative Review 


Systematic Review 


Sources of 
literature 


Non-exhaustive and not stated 
always. Prone to publication 
bias. 


List of databases, grey 
literature and other sources 
are considered. 


Selection criteria 


Usually subjective or no 
selection criteria. Prone to 


Selection process usually 
clear and explicit 


selection bias. 
Appraisal of Variation in evaluation of study | Use of standard checklists 
study quality quality of studies for rigorous appraisal of 
study quality 
Extracting Not explicit and clear Clear and specific 
relevant 
information 
Summary based on studies Clear summaries of studies 
Synthesis which have not been checked | based on high quality 
for quality and can be evidence Narrative, 
influenced by the reviewers quantitative or qualitative 
needs and beliefs synthesis 
Conclusions Sometimes evidence based but | Evidence-based 
could be prone to researcher 
bias (influence of author’s 
personal belief) 
Timeline Weeks to months Months to years 
Requirements Understanding of topic, and At least one of the authors 
searching of 2-3 databases with good knowledge of 
the topic, searches done for 
all relevant databases 
Value Provides summary of literature | Provides high-quality 


on the topic 


Conclusions may be subjective 
hence minimal reproducibility 
of findings 


Cannot be continuously 
updated 


evidence, and supports 
evidence-based practice 


Detailed and accurate 
documentation of methods 
using PRISMA means 
results can be reproduced 
Periodically updated to 
include new evidence 
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Table 2: Characteristics of narrative reviews, scoping reviews and 
systematic reviews 


A priori review No Yes (some) Yes 
protocol 


PROSPERO 
registration of the No No? Yes 
review protocol 


Explicit, transparent, No Yes Yes 
peer reviewed search 
strategy 


Standardized data 
extraction forms No Yes Yes 


Mandatory Critical 
Appraisal (Risk of No No? Yes 
Bias Assessment) 


Synthesis of findings 
from individual No No Yes‘ 
studies and the 
generation of 
‘summary findings’ 


* Current situation; this may change in time. 

’Critical appraisal is not mandatory; however, reviewers may decide to assess and report 
the risk of bias in scoping reviews. 

‘by using statistical Meta-Analysis (for quantitative effectiveness, or prevalence or 
incidence, diagnostic accuracy, aetiology or risk, prognostic or psychometric data), or 
meta-synthesis (experiential or expert opinion data) or both in mixed method reviews’. 
1.4 Where to find systematic reviews? 

Before undertaking a systematic review, it is necessary to look for ongoing 
reviews or completed reviews on the topic of interest. Below are some useful 
websites to start searching for systematic reviews. 


e Database of Abstracts of Reviews of Effects (DARE) https://www.crd.york.ac.uk/ 
CRDWeb/HomePage.asp 


e Cochrane Database of Systematic reviews 
https://www.cochranelibrary.com/cdsr/reviews 


° National Institute for Health and Care Excellence UK Database of Uncertainties 
about the Effects of Treatments (DUETs) ; https://www.evidence.nhs.uk/ 


Introduction 


National Institute for Health Research - Health technology Assessment NIHR- 
HTA 


https://www.nihr.ac.uk/explore-nihr/funding-programmes/health-technology- 
assessment.htm 


Campbell Library of Systematic Reviews 
https://campbellcollaboration.org/better-evidence.html 


Evidence for policy and Practice Information (EPPI ) Centre - https://eppi.ioe. 
ac.uk/webdatabases4/Intro.aspx?ID=9 


Database of promoting health effectiveness reviews : https://eppi.ioe.ac.uk/ 
webdatabases4/Intro.aspx?ID=9 


Agency for Healthcare Research and Quality 
https://www.ahrq.gov/research/findings/evidence-based-reports/search.html 


Scottish Intercollegiate Guidelines Network: https://www.sign.ac.uk/our- 
guidelines.html 


BMJ Evidence based medicine:https://ebm.bmj.com/pages/collections/ebm_ 
verdict/ 


PROSPERO International prospective register of systematic reviews 
Turning Research into practice (TRIP): https://www.tripdatabase.com/ 


Search filters in major databases e.g. Medline, EMBASE, PSYCHLIT, CINAHL (for 
search filters see ‘searching for specific study types’ below) 


Summary 


A systematic review is a study that identifies a specific review question, 
identifies relevant studies using a comprehensive search strategy, 
appraises the quality of these studies and summarizes their results in a 
scientific manner 


Systematic reviews can be conducted on many different types of 
primary studies 

Systematic reviews can be used for informing policies that impact 
quality, safety and values of health care 


Systematic reviews provide summary evidence from available literature 
but narrative reviews do not follow scientific review methodology 


2. Getting Started 


Before initiating a systematic review, it is important to consider four main 
aspects in managing the review: 


a) Formation of a review team- A systematic review team should include 
experts with a range of skills including expertise in information 
retrieval, epidemiologist or clinical expert, systematic review methods, 
statistics, and other aspects e.g. health economist if required for cost- 
effectiveness/cost-benefit analysis reviews and qualitative expects for 
research methods where appropriate. 


b) Formation of an advisory group — An advisory group including health 
care professionals, patient representatives, service users and experts in 
research methods who may be consulted at key stages may be necessary 
for the funding agencies. 


c) Timeline- The timelines for completing various evidence synthesis 
activities may vary. However, organizations such as Cochrane and 
Campbell Collaboration suggest completing a review within a year. 


d) Stakeholder engagement- Various studies have emphasized the 
importance of engaging stakeholders to ensure that systematic reviews 
are shaped by members from the policy and practice community 
who would be using them**. Figure 1 provides a conceptual model of 
stakeholders and the actors. 


e) Software- The selection of software will need to be considered for various 
stages of the systematic review process. Table 3 provides a list of some of 
the software applications found useful at various stages of the review. 


Figure 1: Conceptual model of stakeholders, identified by the actors, 
their roles and their actions 


Actors Roles Actions 
Advocacy groups Editors/peer-reviewers Suggest sources of literature 
Business Endorsers Submit articles 
Citizens Evidence holders Undertake the review 
Pocision eniorcers Funders Endorse the review 
Decision-makers Publish Facilitat an F 
Publishers 'ublishers acilitate gece o the review 
Research funders |. | Communicators Ð Read the review 
Researchers Question askers Share the review 
Reviewers Integrate findings into decisions 
Scope influencers Set the review’s methodological 


standards 
Service providers Provide funding and/or in-kind 
contributions 


Service users 


Share knowledge and experience 
of scope and context =, 


Users of the review 


k 


Getting Started 


Example 


Policy maker 


Policy focus 
component of 


» 


deworming 


» 


Uses the review finding 
tojustify policy decisions 
on use of deworming in 
a country 


Table 3: Software for various processes in the systematic review 


Activity Software Cost Organisation 
Reference Zotero Free RRCHNM 
management Mendeley Paid Elsevier 
EndNote Paid Clarivate Analytics 
Reference Manager | Paid Thomson Reuters 
EPPI Reviewer Paid EPPI Centre 
Screening Covidence Free/Paid Covidence 
EPPI Reviewer Paid EPPI Centre 
Rayyan Free Qatar Foundation 
MS-Excels Free Microsoft 
SUMARI Free JBI 
DistillerSR Free for Evidence Partners 
students, 4 
EPPI Reviewer sat EPPI Centre 
Paid 
Coding EPPI Reviewer Paid EPPI Centre 
MS-Excel Free Microsoft 
SUMARI Free JBI 
Data extraction | EPPI Reviewer Paid EPPI Centre 
MS-Excel$ Free Microsoft 
SRDR Free CEBM 
SUMARI Free JBI 
DistillerSR Free for | Evidence Partners 
students, 4 
months 
Critical EPPI Reviewer Paid EPPI Centre 
Appraisal MS-Excel Free Microsoft 
QARI Free JBI 
MAStARI Free JBI 
ACTUARI Free JBI 
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Activity Software Cost Organisation 
Meta-Analysis | RevMan Free Cochrane 
STATA Paid STATA Corp 
SPSS Paid SPSS Inc 
ei A Free CRAN 
MedCalc Paid CMA Corp 
Mix 2.0 Paid MedCalc Inc 
OpenMeta Analyst Paid Biostat XL 
MAStARI Free CEBM 
Free JBI 
Qualitative nVivo Paid QSR International 
QARI Free JBI 
Mixed Methods | MAXQDA Paid VERBI GmbH 


*MS-Excel would need to be formatted to conduct the various components of the review 


process. JBI softwares are available only for reviews with the Joanna Briggs Institute 
(JBI), Australia. 


The following sites have some learning resources for beginners: 


Campbell Collaboration: https://www.campbellcollaboration.org/ 

Cochrane: https://www.cochrane.org/ 

EPPI-Centre: https://eppi.ioe.ac.uk/cms/ 

Systematic Review Toolbox: http://systematicreviewtools.com/ 

International Initiative for Impact Evaluation (3ie): https://www.3ieimpact.org/ 
Centre for Evidence Based Medicine https://www.cebm.net/ 


Centre for Reviews and Dissemination Databases: https://www.crd.york.ac.uk/ 
crdweb/ 


Health Evidence Network (HEN), WHO: https://www.euro.who.int/en/data-and- 
evidence/evidence-informed-policy-making/health-evidence-network-hen 


Africa Evidence Network: www. https://www.africaevidencenetwork.org/en/ 
The EQUTOR Network: https://www.equator-network.org/ 

The GRADE Working Group: www. hitps://gradeworkinggroup.org/ 

NIHR HTA: https://www.journalslibrary.nihr.ac.uk/#/ 


Getting Started 


2.1 Writing a Systematic Review Protocol 


A review protocol is a guide for a well written systematic review. It explains 
the rationale for conducting the systematic review, states the hypothesis and 
outlines the methodology to be used. It is important to note that the protocol 
is a priori statement of aims and methods of the systematic review, and 
referred back to whenever is needed during the systematic review process. 
Research question(s), aims and methods are considered in advance to identify 
the relevant literature to ensure the conduct of the review with minimal 
bias, access to peer review, greater efficiency in review process®. Protocol 
development is often an iterative process that requires discussion within the 
review team, advisory group and sometimes with the funding agency. Peer 
review and publication makes the protocol publicly available. 


A Cochrane review protocol is considered as an individual publication. Non- 
Cochrane protocols should be registered on PROSPERO - an international 
database of prospectively registered systematic reviews in health and social 
care’. Key features from the review protocol are recorded and maintained 
as a permanent record. Systematic reviews should be registered at inception 
(i.e. at the protocol stage) to help avoid unplanned duplication and to enable 
the comparison of reported review methods with what was planned in the 
protocol’. This prevents duplication (research waste) and makes the process 
easy when the full systematic review is sent for publication. 


By writing a protocol and adhering to it during the review process, the 
researcher makes it clear that the decisions taken while conducting the 
review are not arbitrary, the decision to include or exclude studies in the 
review are not guided by individual choices or prejudices (bias) of the 
authors or prior knowledge about their results. There are several resources 
available for the beginners such as the Cochrane Handbook’, PRISMA 
Extension for protocols (PRISMA-P)’’, Institute of Medicine-Standards for 
systematic reviews" etc. 


Systematic Review should be undertaken by a team of individuals with 
different areas of expertise. The team should consist of a person with clinical 
expertise, a person with systematic review experience, a methods person 
with statistical expertise, and someone with multidisciplinary experience. 
Someone from the relevant location/population allows double-checking of 
inclusion of studies and data collection. 


2.2 Components of a Systematic Review protocol 
The protocol for a systematic review is written using the following format. 
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Background: This section for a Cochrane systematic review has a structured 
format. Under the sub-heading Description of the condition, the health 
condition/disease under consideration is described with definitions and 
epidemiological information as evidenced from latest research. 


Description of the intervention (for intervention reviews) has a description 
of the intervention to be evaluated in the systematic review; recent 
publications on this intervention from the literature should be aligned to 
the review. The mechanism of action of the intervention is described in a 
paragraph: How the intervention might work, followed by justification 
for conducting the systematic review under: why it is important to do 
this review. In this section the authors are required to mention about other 
published systematic reviews on the topic, if there are any, the gaps therein 
that would be addressed in the present review. 


Objectives: The primary and secondary objectives of the review are stated 
under objectives. Additionally, the pre-specified sub-group analyses within 
the major comparisons need to be listed here. 


Methods: This section of the protocol elaborates on the criteria for 
considering studies for the review (inclusion/exclusion criteria): types of 
studies (randomized, quasi-randomized, cluster-randomized in intervention 
reviews), types of participants (the study population, age groups, sex, 
gestation etc. need to be described) types of intervention (the formulation, 
mode of administration, doses etc. need to be mentioned) and types of 
outcome measures: Primary, secondary under different comparison should 
be described, The procedure to be followed for screening of titles abstracts 
and full text articles requires that all the steps are completed by two persons. 
Incase of disagreements, a third person (usually more experienced) involved 
as a team member is approached for resolution. Reviewers may choose to 
perform a blinded review of the articles retrieved through search. 


Search Methods for identification of studies: A comprehensive and up 
to date and reproducible search is a hallmark of a systematic review. This 
includes electronic Searches as well as searching other resources (hand 
searching, non-indexed journals, conference proceedings and grey literature, 
cross-references/citation searching/manufacturers/personal contacts). It is a 
pre-requisite that at least 2-3 electronic databases are searched, in order to 
qualify as a systematic review. The reviewers need to decide what databases 
(MEDLINE via Pub med, EMBASE, Cochrane central register of controlled 
trials (CENTRAL) Cumulative Index to Nursing and Allied Health Literature 
(CINAHL) and sources will be searched, in the context of their topic. The 
time period for search needs to be specified, search terms and key words 
have to be written and a search strategy needs to be written down. Search is 
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a complex activity in the conduct of a systematic review, requiring skills as 
well as access to databases, and resources. It is recommended to seek help 
from an information specialist/librarian. Cochrane review groups assist the 
review authors with searching through their designated search coordinators. 
It is advisable to consider each of the components of the Participants, 
Interventions, Comparator/comparison and Outcomes (PICO) to derive the 
search terms. It is important to mention how the reviewers would search for 
unpublished data (grey literature), conference abstracts. The name of the 
person who will run the searches, if known, should also be mentioned. 


Searching regional databases (e.g. IndMed), clinical trial registries (e.g. 
CTRI), hand-searching of non-indexed journals, conference proceedings 
and unpublished (grey) literature should be attempted by the reviewers and 
described in the methodology. 


Data Collection and Analysis: This section comprises of the following sub- 
headings and their description. 


Selection of studies: In a systematic review all steps are performed by two 
reviewers independently. This is crucial for avoiding personal biases at any 
step of the conduct of the review. Selection of studies is done as per the 
laid down inclusion exclusion criteria. Two review authors independently 
review the titles and abstracts of articles identified by searches for eligibility. 
Studies are classified as included, excluded or unclear. Full articles are 
retrieved after title and abstract screening to evaluate whether the study 
should be included or not based on the PICO of the review. Disagreements 
between the two reviewers are resolved by discussion or consultation with 
the third reviewer. 


Data Extraction and management: Tworeview authors should independently 
extract data from the included studies on a predesigned and pretested data 
extraction sheet. Authors of the original studies may be contacted in case 
of incomplete information in the published article included in the review. 
The data to be extracted include general information (study ID, date of 
extraction, title, authors, and source of study if not published); study 
characteristics (study design, participants and inclusion/exclusion criteria 
used in the study); details of interventions (Including doses, treatment 
duration, comparison details, and duration of follow up). 


Assessment of Risk of Bias in included studies: ‘Risk of bias assessment 
tool and criteria are described in the Cochrane Handbook for Systematic Reviews 
of Interventions and are used to assess risk of bias for included studies. Two 
review authors should independently assess risk of bias in the included 
studies by assessing randomisation sequence generation; allocation 
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concealment; blinding of participants, personnel, and outcome assessors; 
incomplete outcome data; selective outcome reporting; and other sources 
of bias. 


Measures of Treatment effect: Methods of statistical analyses are detailed 
under this sub-heading. The Cochrane Handbook provides a detailed 
description of the analytical methods to be used for dichotomous viz. risk 
ratios (RRs) and risk differences (RDs) with 95% confidence intervals (CIs). 
For continuous outcomes, measures of effect as weighted mean differences 
(MDs) with 95% CIs and standardised mean difference (SMD) should be 
reported. 


Unit of analysis issues: Analyses should consider the level at which 
randomisation was done -individual or cluster. In the event that cluster- 
randomised studies are going to be included appropriate adjustment for 
clustering would be required (multiply the standard error derived from the 
confidence interval of the effect estimate by the square root of the design 
effect). The generic inverse variance method in Review Manager 5.3 software 
can perform Meta-Analysis using inflated variances. 


Dealing with missing data: Authors of original trials included in the review 
should be contacted in case of incomplete/missing data 


Assessment of heterogeneity: Statistical heterogeneity can be assessed via 
visual inspection of forest plots of included trials, using the æ? test and the I? 
statistic. Trial characteristics (participants, design, interventions, outcomes, 
and risk of bias) are examined to identify the source of any observed 
heterogeneity. There are cut-offs recommended by Cochrane review groups 
for results of the P test: < 25% none, 25% to 49% low, 50% to 74% moderate, 
and 75%+ high heterogeneity”. 


Assessment of reporting biases: Reporting biases are assessed by trying 
to identify whether the study was included in a trial registry, whether a 
protocol was published, and whether the methods section provides a list 
of outcomes. The reported outcomes can be compared with what has been 
mentioned in the trial protocol by the trial authors versus the outcomes 
reported in the published article. 


Data Synthesis: In this section the systematic reviewers describe the 
statistical methods of combining data from included studies in a Meta- 
Analysis if possible. However, the studies should be similar, to be combined 
in the Meta-Analysis. Statistical guidelines are available in the Cochrane 
Handbook for Systematic Reviews of Interventions’. The Rev-Man software”, 
free to download can be used for conducting the analyses. A fixed-effect 
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or a random-effects model may be chosen as appropriate. In case of high 
heterogeneity Meta-Analysis is not recommended, only anarrative summary 
of trial findings may be provided. 


Risk of bias: Each individual study included in a systematic review 
should be assessed for key sources of bias. Selection bias could occur due 
to systematic differences in baseline characteristics between the groups 
compared ina study, or in randomized trials, from an inadequate generation 
of a random allocation sequence or inadequate concealment of allocations 
before group assignment. Other biases such as detection bias, performance 
bias, attrition bias and outcome reporting bias can also occur. It is important 
that review authors report the methods used to assess the risk of bias in 
individual studies, as well as the findings of the assessment. The Cochrane 
Collaboration’s tool for assessing the risk of bias in a systematic review, 
each study is graded as low (-), high (+) or unclear (?) across different 
types of bias using a domain-based qualitative description of critical areas 
of potential bias in clinical trials. For meta-analyses, authors can conduct 
sensitivity analyses that exclude trials at high risk of bias to determine the 
effect on the results. 


Quality of evidence: GRADE approach, as outlined in the GRADE Handbook, 
is used to assess the quality of evidence for the clinically relevant outcomes". 
Two review authors independently assess the quality of evidence for each 
outcome. Evidence from RCTs is initially considered high but may be 
downgraded one level for serious (or two levels for very serious) limitations 
on the basis of the following: design (risk of bias), consistency across studies, 
directness of evidence, precision of estimates, and presence of publication 
bias. A software ‘GRADE pro GDT is used to create a ‘Summary of findings’ 
table to report the quality of the evidence. The GRADE approach yields an 
assessment of the quality of a body of evidence according to one of four 
grades. 1. High: We are very confident that the true effect lies close to that 
of the estimate of the effect. 2. Moderate: We are moderately confident in 
the effect estimate: The true effect is likely to be close to the estimate of 
the effect, but there is a possibility that it is substantially different. 3. Low: 
Our confidence in the effect estimate is limited: The true effect may be 
substantially different from the estimate of the effect. 4. Very low: We have 
very little confidence in the effect estimate: The true effect is likely to be 
substantially different from the estimate of effect. 


Subgroup analysis and investigation of heterogeneity: Sub-group analyses 
are conducted to explore the reason for heterogeneity detected. Review 
authors should a priori determine and describe the possible sub-groups in 
their review under the comparisons envisaged. 
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Sensitivity analysis: Sensitivity analyses are conducted to assess the impact 
of high risk of bias on the outcome of meta-analyses by adding studies with 
high risk of bias to pooled studies with low risk of bias. Similarly, other 
assumptions can also be changed to see their changing effect on the overall 
estimate. 


References: A list of references of studies included, excluded in the review 
and additional references used in the background section should be 
provided. The study ID usually comprises of the Surname of the author and 
the year of publication. 


Acknowledgement: Authors may list the names of people who have helped 
them in the process of conducting the systematic review but who do not 
qualify as authors. 


Appendices: The detailed search strategy is usually included in this 
section 


Contribution of Authors: The contribution of each review author should be 
mentioned in this section. 


Declaration of interest: Any conflicts of interests should be stated in this 
section 


Sources of Support: If the authors have received funding to conduct 
the systematic review, they should mention it here. If not, they should 
acknowledge the internal support of their respective organizations. 


Example of a Systematic Review Protocol: Sinha A, Pradhan A, Thumburu 
KK, Gupta N. Probiotics for the prevention or treatment ofhyperbilirubinemia 
in late preterm and term neonates. Cochrane Database of Systematic Reviews 
2017, Issue 8. Art. No.: CD012781. DOI: 10.1002/14651858.CD012781 
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3. Steps in a systematic review 


A systematic review is based on a clearly formulated question, identifies 
relevant studies, appraises the quality and summarizes the evidence by an 
explicit methodology. Figure 2 shows the common stages in a systematic 
review. 


Figure 2: Common stages of a systematic review 


Review initiation 
Form review team Engage stakeholders 


| 


Review question & methodology 
Formulate question, conceptual framework & approach 


| 


Search Strategy | 


Search & screen for inclusion using eligibility criteria 


| 


Description of study characteristics 
Code to match or build a conceptual framework 


| 


Quality and relevance assessment 
Apply quality appraisal criteria 


| 


Synthesis 
Use conceptual framework, study codes & quality judgments 


Reporting 
Interpret & communicate findings with stakeholders 


(Adapted from Gough, Oliver and Thomas, 201215) 
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3.1 Structure an answerable and focussed review question 


Formulating a research question is the most critical and difficult part of 
any research design. The review question underpins all the aspects of the 
review methodology and every single step of the review is determined by 
the focussed review question. A review question defines the nature and 
scope of the review, identifies the keywords, determines the search strategy, 
provides guidance for selecting the primary research papers and guides the 
data extraction and synthesis of results. When formulating a review question, 
it is important to ensure that the question asked is an open question and not 
a statement. For example, rather than saying that “antiseptic washes prevent 
nosocomial infections in patients undergoing surgery “it would be better 
to ask “Are antiseptic washes more effective than non-antiseptic washes at 
preventing nosocomial infections in patients undergoing surgery?”. Table 
4 lists some review questions that guide in identifying primary research 
papers for a systematic review. 


Table 4: Types of review questions 


Type Description Example 
Treatment | Which treatment is more | Is hydrocolloid occlusive dressing 
or therapy | effective? Does it do better | better than conventional gauze 
than harm? dressing in the healing of chronic 
wounds? 
Prevention | How to reduce the risk of | Does increasing physical activity 
disease? reduce the risk of developing 
diabetes? 
Diagnosis | How to select and interpret | Is MRI scan more effective than X ray 
diagnostic tests? in identifying hairline fractures? 
Prognosis | Howtoanticipate the likely | Are babies who are bottle fed more 
course of the disease? likely to be obese in adulthood 
compared to babies who are breast 
fed? 
Causation | What are the risk factors | Does exposure to smoking in mothers 
for developing a certain | who smoke during pregnancy increase 
condition? risk of foetal death? 


Once a tentative review question is formed it is essential to structure it into 
a well framed structured question that includes three or four elements. 
A structured question includes the population, the intervention, the 
comparative intervention and the outcomes that are measured. The acronym 
for this is PICO which stands for Population, Intervention, Comparator, and 
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Outcome. Depending on the type of study design there are variants of the 
acronym for eg. PEO for patient exposure and outcome, PICOC Population, 


Intervention, Comparator/s, Outcomes, Context etc. 


Depending on the 


study design, the components of the review question are accordingly 
identified. Table 5 provides a list of review questions and the corresponding 
components of the research question. 


Table 5- The PICO process 


Question | Patient problem Intervention | Comparison Outcome 
type or exposure measures 
Treatment | The patient’s A therapeutic | Standardof | Mortality 
(Therapy) disease or measure for care, another | rate, work 
condition eg. surgical intervention | days lost, 
intervention or placebo pain, 
or life style disability 
change 
Prevention | The patients A preventive | May notbe Disease 
risk factors and | measure, drug | applicable incidence, 
general health or life style mortality 
condition change rate, work 
days lost 
Diagnosis | Target disease or | A diagnostic Current Measure of 
condition test or reference test utility, 
procedure standard or sensitivity, 
gold standard | specificity, 
test for the odds ratio 
problem 
Prognosis The main Time or Usually not Survival 
(Natural prognostic watchful applicable. rates, 
History) factor or clinical | waiting Sometimes mortality 
problem in the standard | rates, rates 
terms of its treatment of disease 
severity and progression 
duration 
Etiology Risk factors, The May not be Disease 
or harm current health intervention or | applicable incidence, 
(Causation) | problems, the exposure rates of 
general health of interest disease 
condition including progression, 
some mortality 
indication of rates 
strength(dose) 
of the risk 
factor and the 
duration of the 
exposure 
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The templates below and the figure 3 shows how to build research questions 
in different scenario and how the components are related 


THERAPY 

In what is the effect of on 

compared with 2 

PREVENTION 

For does the use of reduce the future risk of 
compared with ? 


DIAGNOSIS OR DIAGNOSTIC TEST 


Are (Is) more accurate in diagnosing 

compared with ? 

PROGNOSIS 

Does influence in patients who have 
? 

ETIOLOGY 

Are who have at 

risk for/of compared with with/ 

without, ? 

EXPERIENTIAL 

How do diagnosed with perceive 
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Figure 3: Structured questions for systematic reviews and relation 
between question components in a comparative study 


The population 
Study sample A clinically suitable group of 
participants or patients 
Allocation of 
participants to groups 
The interventions 


Experimental group Control group Comparison of groups with and 
(e.g. a new intervention) (e.g. usual care) without the intervention 
or exposure of interest 


Outcome Outcome Outcome The outcomes 
present absent absent Changes in health status due 
a b c d to interventions 


Experimental event rate Control event rate 
EER = a/(a+b) CER = c/(c+d) 
The effects 
Estimate of effect 
e.g. relative risk = EER/CER A measure of association between 
interventions and outcomes 


(Adapted from Khan, Kunz, Kleijnen, & Antes, 2003*6) 


Outcome 
present 


Ideally once the research question is stated with the four or five relevant 
elements, the objectives and the title of the systematic review become clear. The 
template in Box 1 provides examples on how to state the problem statement, 
create the review question, and set the objectives and finally form a title for the 
review. 


Box 1-Quantitative review of domestic violence an example 


Problem statement: Little is known on the effectiveness of advocacy 
programmes as compared to other treatments on women’s quality of life 
among those who have experienced domestic violence 


Review question: For women who have experienced domestic violence, 
how effective are advocacy programmes compared to other treatments 
on improving the quality of life 
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Aim: The aim of this study is to evaluate the effectiveness of advocacy 
programmes as compared to other treatments on improving the quality 
of life of those women who have experienced domestic violence 
Objectives: 


* Search papers on effectiveness of advocacy programmes and other 


treatments 


Collect data on the effectiveness of advocacy programmes and other 
treatments 


Compare the primary studies on the different treatments 
Compare the findings of this review with other reviews 


Provide guidelines for helping women who have experienced 
domestic violence 
Title: A systematic review on the effectiveness of advocacy programmes 


as compared to other treatments on improving the quality of life of those 
women who have experienced domestic violence. 


3.2 Types of systematic reviews 


Although systematic reviews are predominantly conducted to assess the 
effectiveness of health interventions, researchers and health professionals 
are also concerned with other questions that need different approaches. 
Forcibly pushing the question into the PICO format (population, 
intervention, comparator and outcome), even though the question may be 
on diagnostic test accuracy or prognosis can confound the remainder of the 
review process. Based on the type of question addressed, about 13 types of 
reviews have been identified by Zachary et al”. Table 6 provides the list of 
different review types. 


3.2.1 Effectiveness reviews 


Effectiveness is the extent to which an intervention has an effect. The PICO 
approach is recommended here to define the population (e.g. demographic 
and socioeconomic, factors and setting), intervention (e.g. variations 
in dosage/intensity, delivery mode, and frequency/duration/ timing of 
delivery), comparator (active or passive) and outcomes (primary and 
secondary) including benefits and harms. 


3.2.2 Experiential (qualitative) reviews 


Experiential reviews are qualitative in nature that focusses on the perspective 
of the individuals’ experience, analysing human experiences and cultural, 
social phenomena. They can be important in exploring and explaining why 
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interventions are or are not effective from a person-centred perspective. With 
qualitative evidence, there is no outcome or comparator to be considered. 
PICO is recommended for question development with P and I defined as 
patient and experience eg. response to pain. Context may be geographic 
location, specific racial or gender or setting such as acute care or primary 
healthcare or community. 


3.2.3 Costs/economic evaluation reviews 


Costs/Economics reviews assess the costs of a certain intervention, process, 
or procedure. Health economic evaluations are useful to inform health 
policy. The PICO approach best fits here with the population, intervention 
and comparator that include the nature of services/care delivered, time 
period of delivery, dosage/intensity, co-interventions. Context can also be 
considered in these types of questions e.g. health setting(s). 


3.2.4 Prevalence and/ or incidence reviews 


Prevalence or incidence reviews measure disease burden. These types of 
reviews inform health care planning and allocation of resources, delivery of 
health services and evaluate changes and trends in diseases over time. The 
CoCoPop framework can be used for such reviews. The health condition, 
disease, symptom, and its measurement, diagnosis needs to be identified. 
Environmental factors define the context or specific setting relevant to the 
review question 


3.2.5 Diagnostic test accuracy reviews 


Systematic reviews assessing diagnostic test accuracy are important for 
clinicians and other healthcare practitioners to determine the accuracy of the 
diagnostic tests to identify the presence or absence of a condition for treatment 
plan in a patient. For these review questions, PIRD is recommended where 
the population consists of all participants who will undergo the diagnostic 
test. It could be a comparison of this test with the gold standard, for the 
diagnosis of the intended condition viz disease, disability or injury. 


3.2.6 Aetiology and/ or risk reviews 


Systematic reviews of aetiology and risk are important for making health 
policy decisions and prevention of adverse health outcomes. These reviews 
determine whether and to what degree a relationship exists between an 
exposure and a health outcome. PEO is recommended for these types of 
review questions. The risk factor associated with the condition, the dose 
and nature of the exposure, the duration of exposure, disease, symptom or 
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health condition, the population at risk, the context/location, are relevant in 
such reviews. The outcomes of interest include the health policy issues. 


3.2.7 Expert opinion/ policy reviews 


Expert opinion and policy analysis systematic reviews focus on the synthesis 
of narrative text and/or policy to either complement empirical evidence or, 
in the absence of research studies, stand alone as the best available evidence. 
In the absence of research studies, the best available evidence can be drawn 
from text and opinion to guide researchers and policy makers. PICo can be 
used where reviewers need to describe the characteristics of the population, 
such as age, gender, level of education or professional qualification, 
interventions may be areas of practice management. The use of a comparator 
and outcome is not required. 


3.2.8 Psychometric reviews 


Psychometric systematic reviews are conducted to assess the quality in 
terms of its validity, reliability, responsiveness etc of health measurement 
instruments for a specific construct. The construct, name of the outcome 
measurement, the target population, the type of measurement instrument of 
interest (e.g. questionnaires, imaging tests) and the measurement properties 
on which the review investigates, needs to be clearly specified. 


3.2.9 Prognostic reviews 


Prognostic research provides information on the course of a disease and 
potential outcomes. These reviews should identify the relationship between 
specific prognostic factors and an outcome and/or prognostic/prediction 
models and prognostic tests. 


3.2.10 Methodology systematic reviews 


Methodology Systematic reviews are performed to examine any 
methodological issues relating to the design, conduct and review of research 
studies and also evidence syntheses. The types of studies (RCTs and quasi- 
RCTs), the comparisons of interest and the primary and secondary outcome 
measures should be identified. 


3.2.11 Implementation reviews 


Systematic reviews of implementation research studies are performed to 
study the strategies used to integrate evidence based practices to real world 
settings. 
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3.2.12 Predictors reviews 


Reviews of studies to identify predictors of poor clinical outcomes, 
improvement in health and social outcomes, quality of life, behavioural 
changes etc. are conducted by synthesising evidence from observational 
studies 


3.2.13 Barriers and facilitators reviews 


Reviews that synthesize research on barriers to and facilitators of the various 
health outcomes. It provides an overview of factors related to uptake and 
implementation of a health intervention and health promoting behaviours. 
These reviews help the policy makers in making efforts to overcome the 
barriers and promote facilitating factors. 


Table 6: Types of reviews 


Aim Question Format | Type of Example 
review 
To evaluate the Population, Effectiveness | What is the 
effectiveness of a Intervention, effectiveness 
certain treatment/ Comparators, of exercise for 
practice in terms of its | Outcomes treating diabetes 
impact on outcomes (PICO) compared to 
no treatment 
or comparison 
treatment? 
To investigate Population, Experiential | What is 
the experience or Phenomena of (Qualitative) | experience of 
meaningfulness of a Interest, Context children with 
particular phenomenon | (PICo) epilepsy? 
To determine the Population, Costs/ What is the cost- 
costs associated with Intervention, Economic effectiveness of 
a particular approach/ | Comparator/s, Evaluation HIV vaccines 
treatment strategy, Outcomes, in low- and 
particularly in terms Context (PICOC) middle-income 
of cost-effectiveness or countries? 
benefit 
To determine the Condition, Prevalence What is the 
prevalence and/or Context, and/or prevalence 
incidence of a certain Population (Co Incidence of diabetes in 
condition CoPop) India? 
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Aim Question Format Type of Example 
review 

To determine how Population, Diagnostic What is the 

well a diagnostic Index Test, Test Accuracy | diagnostic test 

test works in terms Reference Test, accuracy of 

of its sensitivity Diagnosis of nutritional tools 

and specificity for a Interest (PIRD) (such as the 

particular diagnosis Malnutrition 
Screening Tool) 
compared to 
the Patient 
Generated 
Subjective 
Global 
Assessment 
amongst 
patients with 
colorectal cancer 
to identify 
undernutrition? 

To determine the Population, Etiology and/ | Are adults 

association between Exposure, or Risk exposed to 

particular exposures/risk | Outcome (PEO) arsenic at risk of 

factors and outcomes lung cancer? 

To review and Population, Expert What are 

synthesize current intervention or opinion/ the national 

expert opinion, text Phenomena of policy policies to 

or policy on a certain Interest, Context reduce maternal 

phenomenon (PICO) mortality? 

To evaluate the Construct of Psychometric | What is the 

psychometric interest or the reliability, 

properties of acertain | name of the validity, 

test, normally to measurement (s), responsiveness 

determine how the Population, Type and 

reliability and validity | of measurement, interpretability 

of a particular test or instrument, of methods 

assessment Measurement (manual 

properties muscle testing, 
(CoPoTIM) isokinetic 

dynamometry, 
hand held 
dynamometry) 


to assess muscle 
strength in 
adults? 
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Aim Question Format Type of Example 
review 

To determine the Population, Prognostic In adults with 

overall prognosis Prognostic low back pain, 

for a condition, the Factors (or what is the 

link between specific models of association 

prognostic factors and | interest), between 

an outcome and/or Outcome (PFO) individual 

prognostic/prediction recovery 

models and prognostic expectations and 

tests disability 
outcomes? 

To examine and Types of Methodology | Organizational 

investigate current studies, Types ethics research: 

research methods of Data, Types A systematic 

and potentially their of Methods, review of 

impact on research Outcomes methods and 

quality (SDMO) analytical 
techniques 

To evaluate the factors | SCOOPS (setting, | Implementa- | Systematic 

that are associated condition/ tion review on 

with successful/failure | circumstance, implementation 

of implementation output, outcome, of mobile health 

programs /projects/ process, strategy) (mHealth) 

interventions projects in 
Africa: What 
works? What 
doesn’t work 
and why? 

To identify the factors Condition, Predictors What are the 

associated with a Predictor, childhood 

certain condition Population predictors of 

(CoPrePop) adult obesity? 

To identify the factors | Population, Barriers and | Barriers and 

that are barriers and/or | Factors (Barriers/ | facilitators facilitators to 

facilitators for access/ Facilitators), health screening 

uptake of a program Intervention in men: A 

(PFal) systematic 

review 


3.3 Identify relevant studies through a comprehensive search 


strategy 


The search strategy used in the review ideally should follow established 
guidelines such as MECIR (Methodological Expectations of Cochrane 
Intervention Reviews) or MECCIR (Methodological Expectations of 
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Campbell Collaboration Reviews)’. As per the Cochrane Handbook; 
‘Systematic reviews of interventions require a thorough, objective and 
reproducible search of arange of sources to identify as many relevant studies 
as possible. This is a major factor in distinguishing systematic reviews 
from narrative reviews and helps to minimize bias and therefore assist in 
achieving reliable estimates of effects. 


When a systematic review is conducted, it is important to retrieve all the 
relevant studies both published and unpublished that may answer the 
proposed research question. A comprehensive search strategy underlies the 
quality of search and the quality of findings of the review. A search has 
to be both sensitive and specific. Sensitive means the search that picks up 
all the research studies that are potentially relevant and specific means, it 
selects only those that are directly relevant. 


Step 1 Identify the component parts in the review question ie the P, I, C, O. 


The previous section discussed in detail on the structure of a review question 
using these components. 


Step 2 Identify any keywords and synonyms 


The second step is to identify keywords and synonyms for all component 
parts of the review question. Keywords describe subject areas, and having 
a good set of keywords, would minimize the number of irrelevant returns. 
Keywords can be identified using dictionaries, textbooks, lecture notes and 
published articles. Synonyms are words which have the same or similar 
meaning. Authors use different words and phrases to describe subject areas. 
It is necessary to ensure that all synonyms and keywords are included in 
the search strategy to avoid missing potentially relevant documents. Box 2 
provides an example of key words and the synonyms. 


Box 2: List of keywords and synonyms related to a question 


Terms that have the same/close Hypertension vs High blood pressure 

meaning 

Terms that have different spellings or | Leukemia vs Leukaemia 

acronyms 

Complex concepts described Long-term patient-reported satisfaction after 

inconsistently contralateral prophylactic mastectomy ... vs 
Surviving breast cancer: women’s experiences with 
their changed bodies 

Umbrella terms and specific names Sexually-transmitted infections 
Herpes, genital warts, syphilis, gonorrhoea, chlamydia 

Keywords and database-specific Cancer, tumour, tumour, carcinoma 

“subject headings” Neoplasms (MESH) 
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Step 3. Construct search strategy string 


To retrieve the most relevant search results, a search string needs to be 
constructed. A search string is a combination of keywords, truncation 
symbols, and Boolean operators that is entered into the search box of a 
library database or search engine. 


For eg: A systematic review to analyse long-term continence disturbance 
after lateral internal sphincterotomy for chronic anal fissure was carried out. 
The PIO component of the research question was 


P= Patients with Chronic Anal Fissure 
I= Lateral internal sphincterotomy 
O=Long term continence disturbance 


The keywords and the synonyms for the components are as given in Box 3 


Box 3: Keywords and synonyms for the systematic review example 


Chronic anal fissure(P) | ‘anal fissure’, ‘fissure-in-ano’, ‘chronic fissure’, 


Lateral internal ‘lateral internal sphincterotomy’, ‘sphincterotomy’, 
sphincterotomy(I) ‘fissurectomy’, ‘advancement flap’, ‘diltiazem’, ‘nitro- 
glycerine’ and ‘botulinum toxin’ 


Long term continence ‘incontinence’, ‘continence disturbance’, ‘accidental bowel 
disturbance(O) motion’, ‘urge incontinence’, ‘liquid incontinence’, ‘faecal 
incontinence’, ‘soilage’, ‘seepage’ 


Boolean Operators 
OR AND 


For each PICO element we combine all alternative terms with an OR 

For e.g. for P the combination would be: #1 anal fissure OR fissure-in-ano OR chronic 
fissure. 

Likewise, we combine the terms for the other two elements, #2 for I and #3 for O. 

In the end we use an AND to incorporate results of all PICO elements i.e. #1 AND#2 
AND #3. 

The truncation wildcard * and? can be used with the terms for a wider search. 
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3.4 Undertake a comprehensive search 


After completing the search strategy string, a comprehensive search should 
be done using data bases and all other sources of information that are most 
relevant to the review question. Sources of information could be online data 
bases, specialist data bases, journal articles, grey literature, subject gateways, 
conference papers and proceedings, dissertation abstracts, contacting the 
experts and books. 


3.5 Select databases 
3.5.1 Published literature- Where to search? 


The data bases for searching published literature are varied and depend on 
the type of review and the context or subject of the systematic review. Box 
4 provides a list of sources. 


Box 4: Sources of published literature 


Core 

Cochrane Library Intervention and diagnostic reviews 

Cochrane Reviews Critically appraised and re-structured abstracts 

Cochrane Protocol Register of clinical trials 

Other reviews 

Trials 

Medline Three different versions: PubMed, OVID Medline 
and EBSCO Medline (Books@ovid, Full text, 
Medline from 1946, and Epub Ahead of print, 
In-Process, In-Data review & other Non-Indexed 
Citations) 

Embase Pharmacological literature, conference abstracts 
Medical devices(from 1947) 

Web of Science Conference abstracts, citation searching, social 
sciences 
Education, Conference Proceedings Citation Index - 
Science (CPCI-S; 1990 onwards). 
Conference Proceedings Citation Index - Social 
Science & Humanities (CPCI-SS&H; 1990 onwards) 

Campbell Systematic | Social sciences 

Reviews 
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SCORUS 


Conference abstracts, citation searching (from 1996), 
patents, scientific webpages, Trade Publications, 
Book series 


India (CTRI) 


Clinicaltrials.gov 
Clinical Trials Registry of 


South Asian Database of 
Controlled Clinical Trials 


Trials registered in US and global 
Trials registered in India 


Controlled clinical trials and their sources in India 
and other South Asian countries 


(SADCCT) 

Indexing of Indian | Indexed selected peer reviewed medical journals 

Medical Journals | published from India. It supplements international 

(IndMED) indexing services like PubMed. It covers about 100 
journals indexed from 1985 onwards. 

Subject/Study 

Dependent 

CINAHL Nursing and allied health 

Psychinfo Psychology & psychiatry 

ERIC Education 

TOXLINE Effects of drugs and chemicals 

PedRO Physiotherapy (randomized controlled trials and 
systematic reviews only) 

PEDE (Pediatric | Paediatric economic evaluations 

Economic Database | inventory of health state utility weights reported in 

Evaluation) cost-utility analyses 

CEA Registry Cost-utility analyses on a wide variety of diseases 


and treatments 


3.5.2 Grey literature- Where to search? 


Grey literature are materials and research produced by organizations outside 
of the traditional commercial or academic publishing and distribution 
channels. These publication types include reports, working papers, 
government documents, white papers and evaluations. Grey literature covers 
published material not indexed in databases such as Medline, Embase etc, 
which index principally journal literature. These include technical reports, 
official publications, conference papers, dissertations, patents, research in 
progress, usually produced by academic, government and professional 
organisations. It is important to search grey literature sources in order to 
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minimise bias in your search results. Box 5 provides a list of sources of grey 
literature. 


Box 5: Sources of grey literature 


Name Note 


Google Scholar Initial background searches 
Include along with PubMed 


Open Grey (www. | System for information on grey literature in Europe 
opengrey.eu) 


Social Science | Specialised research networks (economics, business & 
Research Network | management) in each of the social sciences. Includes 
(http://ssrn.com/) | abstracts database of forthcoming papers and working 
papers as well as Electronic Paper Collection of full text 


documents. 

ProQuest Masters, MPhil and PhD theses database of international 
universities 

Shodhganga Reservoir of theses from Indian universities 


3.6 Tailor search strategy to database(s) 


Different databases will need different search fields for conducting a search: 


e.g. PubMed: 
Search the Text Word field for free-text terms and MeSH Terms for controlled 
vocabulary terms. 

e.g. sunshine[tw], sunlight[tw], Sunbathing[mh], Suntan[mh] 


e.g. Embase: 
Search the Title and Abstract field :ti, ab for free-text terms and 
Emtree terms ‘term’/exp for controlled vocabulary terms. 

e.g. sunshine:ti,ab, sunlight:ti,ab, “sun exposure’/exp, ‘sunlight’/exp 


Syntax for PubMed (From Cochrane Handbook, 6.4.11.1) 
#1 randomized controlled trial [pt] 
#2 controlled clinical trial [pt] 
#3 randomized [tiab] 
#4 placebo [tiab] 
#5 drug therapy [sh] 
#6 randomly [tiab] 
#7 trial [tiab] 
#8 groups [tiab] 
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#9 #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 
#10 animals [mh] NOT humans [mh] 
#11 #9 NOT #10 


3.7 Save search and export results 


It is vital that the search strategy is transparent and repeatable, and hence 
for each source searched, a record should be made (Table 7). The full list of 
results from each search and source must be recorded for transparency and 
can be efficiently managed using reference management software such as 
Endnote, Zotero etc. Many systematic review software such as Covidence, 
EPPI Reviewer etc, allow importing the search results from RIS file directly 
from such reference management software. 


3.8 Select studies for inclusion based on pre-defined criteria 


The relevant articles are screened using the inclusion criteria at different 
levels of reading to impose a number of filters of increasing rigor, i.e. first 
reading of article titles and abstracts to remove spurious hits; and for those 
passing through this stage an assessment of the full text. In order to manage 
errors during the screening process, it is a good practice that two reviewers 
undertake both title/abstract and full-text screening. A kappa analysis can 
be performed to check for consistency in the interpretation of the selection 
criteria between the two reviewers. A kappa rating of ‘substantial’ (0.5 or 
above) is recommended to pass the assessment. It is important that the 
selection of each article captured during the search should be recorded as 
per PRISMA flowchart (see Figure 8). 


Table 7: Saving a search strategy and the number of records 


#1 | (“calcium hydroxide” OR 24834 
(medicament OR “intracanal medicament”) OR 
dressing 

#2 | (irrigation OR irrigating OR irrigated OR irrigant 55869 
OR irrigate OR(rinse OR rinsing OR rinsed) 

#3 | ((remov*) OR (eliminat*) 565298 

#4 (endodontics OR “root canal” OR “dental pulp 129733 
activity” OR teeth OR tooth) 

#5 | #1 AND #2 AND #3 AND #4 156 
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3.9 Description of study characteristics 


Once the full text of each article is identified for inclusion in the systematic 
review, the next step is to extract appropriate and relevant data using a 
standardized data extraction/coding form. Coding instruments, such as 
coding sheets and codebook, are designed for specific research”! synthesis 
and are based on the types of interventions, outcome variables, and other 
data. Brown et al” categorizes that coded data fall into the following four 
basic categories: (1) methodological and substantive features, (2) study 
quality, (3) intervention descriptors, and (4) outcome measures. In order to 
design an accurate and comprehensive coding scheme, there is aneed fora 
thorough knowledge of the included studies that are to be included in the 
synthesis. The important variables identified to be coded in every research 
synthesis, include: (i) Source of the study; (ii) Year of publication; (iii) Type 
of research design. Once the coding sheet is completed, a codebook has to 
be developed to guide the coding process, containing each variable that is 
important, and should be pilot tested. 


Many reviewers extract information on study characteristics, methodology, 
population, interventions and outcomes, with the outcomes varying based 
on the types of study designs included. For example, if RCTs are included, 
the outcomes are usually expressed as risk ratios (RR), odds ratio (OR) 
or difference between means for continuous outcomes; for diagnostic 
studies, the outcomes extracted are the measures of test performance (e.g. 
sensitivity and specificity). Box 6 provides good practice guidelines for 
data extraction from primary studies 


Box 6: Good practice for data extraction 


Good practice could involve the following steps, which improve 
transparency, repeatability and objectivity: 


e Data extractions should always present the primary data as reported 
in the primary study; if any corrections or transformations are needed 
these should be presented additionally so that all data are traceable to 
the primary study 


Notation of the location of data within each article and means of 
extraction if data are located within figures. 


Description of any pre-analysis calculations or data transformations 
(e.g. standard deviation calculation from standard error and sample 
size and calculation of effect sizes. 
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Details of a pre-tested data extraction form. 


Data extraction in a subset of articles by multiple reviewers 
and checking, for example with a kappa test (for human error/ 
consistency) 


Inclusion of appendixes of extracted information 


Contact made with authors requesting data where it is missing from 
relevant articles 


Adapted from Collaboration for Environment Evidence, 2013% 


It is difficult to design a single form that meets a reviewer's requirement. 
A data extraction form should be comprehensive including all the relevant 
information needed for reporting and analysing the data. Below is a data 
extraction form provided by the Cochrane collaboration”. This can be used 
for both RCT and non RCT reviews. 
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Data collection form 


Intervention review — RCTs and non-RCTs 

THE COCHRANE 

COLLABORATION® 
This form can be used as a guide for developing your own data extraction 
form. Sections can be expanded and added, and irrelevant sections can be 
removed. It is difficult to design a single form that meets the needs of all 
reviews, so it is important to consider carefully the information you need 
to collect, and design your form accordingly. The information included on 
this form should be comprehensive, and may be used in the text of your 
review, ‘Characteristics of included studies’ table, risk of bias assessment, 
and statistical analysis. 


Notes on using a data extraction form: 


e Be consistent in the order and style you use to describe the information 
for each included study. 


e Record any missing information as unclear or not described, to make it 
clear that the information was not found in the study report(s), not that 
you forgot to extract it. 

e Include any instructions and decision rules on the data collection form, or 
in an accompanying document. It is important to practice using the form 
and give training to any other authors using the form. 

e You will need to protect the document in order to use the form fields (Tools 
/ Protect document) 


Review title or ID 


Study ID (surname of first author and year first full report of study was published e.g. 
Smith 2001) 


Report IDs of other reports of this study (e.g. duplicate publications, follow-up 
studies) 


Notes: 


1. General Information 


Date form completed (dd/mm/yyyy) 


1. 
2. | Name/ID of person extracting data 
3. 


Report title 
(title of paper/ abstract/ report that data are extracted from) 
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4. Report ID 
(if there are multiple reports of this study) 


5. Reference details 


6. Report author contact details 


7. Publication type 
(e.g. full report, abstract, letter) 


8. Study funding source 
(including role of funders) 


Possible conflicts of interest 
(for study authors) 


9. Notes: 


2. Eligibility 


Study Review Inclusion Criteria Location 
Characteristics (Insert inclusion criteria for each Yes/ in text 
characteristic as defined in the Protocol) No/ pey 
Unclear | fig/table) 
10. Type of Randomised trial 
study 


Non-randomised trial 


Controlled before-after study 
e Contemporaneous data 
collection 
e At least 2 intervention and 2 
control clusters 


Interrupted time series OR 
Repeated measures study 
e At least 3 timepoints before 
and 
3 after the intervention 
e Clearly defined intervention 
point 


Other design (specify): 


11. Participants 


12. Types of 
intervention 
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Study Review Inclusion Criteria Location 
Characteristics (Insert inclusion criteria for each Yes/ in text 
characteristic as defined in the Protocol) No/ pey 
Unclear | fig/table) 
13. Types of 
outcome 
measures 
14. Decision: 
15. Reason for 
exclusion 
16. Notes: 


DO NOT PROCEED IF STUDY EXCLUDED FROM REVIEW 


3. Population and setting 


Description 

Include comparative information for each 
group (i.e. intervention and controls) if 
available 


Location in 
text 

(pg & Wfig! 
table) 


17. Population 

description 
(from which study 
participants are 
drawn) 


18. Setting 
(including location 
and social context) 


19. Inclusion 
criteria 


20. Exclusion 
criteria 


21. Method/s of 
recruitment of 
participants 


22. Notes: 


36 


Steps in a systematic review 


4. Methods 
Descriptions as stated in report/ | Location in 
paper text 
(ps & Whig! 
table) 


23. Aim of study 


24. Design 
(e.g. parallel, crossover, 


non-RCT) 


25. Unit of allocation 
(by individuals, cluster/ 
groups or body parts) 


26. Start date 


27. End date 


28. Duration of 
participation 
(from recruitment to 

last follow-up) 


29. Notes: 


5. Risk of Bias assessment 


See Chapter 8 of the Cochrane Handbook. Additional domains may be required for 
non-randomised studies. 


Domain Risk of bias Support for Location in text 
Low/ High/ judgement (pe & Y/fig/table) 
Unclear 


30. Random sequence 
generation 
(selection bias) 


31. Allocation 
concealment 
(selection bias) 


32. Blinding of Outcome 
ee and 7 group: All/ 
personne 


(performance bias) 
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Domain Risk of bias Support for Location in text 
Low/ High/ judgement (pg & Yfig/table) 
Unclear 
Outcome 
(if required) group: 
33. Blinding of outcome Outcome 
assessment group: All/ 
(detection bias) 
Outcome 
(if required) group: 


34. Incomplete outcome 
data 
(attrition bias) 


35. Selective 
reporting? 
(reporting bias) 


outcome 


36. Other bias 
37. Notes: 


6. Participants 


Provide overall data and, if available, comparative data for each intervention 


or comparison group. 


Description as stated in 
report/paper 


Location in 
text 
(pg & T/fig/ 
table) 


NRCITs) 


38. Total no. randomised 
(or total pop. at start of study for 


39. Clusters 


cluster) 


(if applicable, no., type, no. people per 


40. Baseline imbalances 


41. Withdrawals and exclusions 
(if not provided below by outcome) 


42. Age 
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Description as stated in 
report/paper 


Location in 
text 
(pg & T/fig/ 
table) 


43. Sex 


44. Race/Ethnicity 


45. Severity of illness 


46. Co-morbidities 


47. Other treatment received 
(additional to study intervention) 


48. Other relevant 
sociodemographics 


49. Subgroups measured 


50. Subgroups reported 


51. Notes: 


7. Intervention groups 


Copy and paste table for each intervention and comparison group 


7.1 Intervention Group 1 


Description as stated in 
report/paper 


Location in 
text 

(pg & Wfig/ 
table) 


52. Group name 


53. No. randomised to group 
(specify whether no. people or clusters) 


54. Description 

(include sufficient detail for replication, 
e.g. content, dose, components; if it 1s 
a natural experiment, describe the pre- 
intervention) 


55. Duration of treatment period 


56. Timing 
(e.g. frequency, duration of each 
episode) 
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Description as stated in 
report/paper 


Location in 
text 

(pg & Wfig/ 
table) 


57. Delivery 
(e.g. mechanism, medium, intensity, 
Cc ` 


fidelity) 


58. Providers 
(e.g. no., profession, training, ethnicity 
etc. if relevant) 


59. Co-interventions 


60. Economic variables 
(i.e. intervention cost, changes in other 
costs as result of intervention) 


61. Resource requirements to 
replicate intervention 

(e.g. staff numbers, cold chain, 

equipment) 


62. Notes: 


8. Outcomes 


Copy and paste table for each outcome. 


Outcome 1 


Description as stated in report/ 


paper 


Location in 
text 

(pg & Wfig/ 
table) 


63. Outcome name 


64. Time points measured 
(specify whether from start or 
end of intervention) 


65. Time points reported 


66. Outcome definition 
(with diagnostic criteria if 
relevant and note whether 
the outcome is desirable or 
undesirable if this is not 
obvious) 
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Description as stated in report/ 
paper 


Location in 
text 

(pg & Wig! 
table) 


67. Person measuring/ 
reporting 


68. Unit of measurement 
(if relevant) 


69. Scales: upper and 
lower limits 

(indicate whether high or low 

score is good) 


70. Is outcome/tool 
validated? 


Yes/No/Unclear 


71. Imputation of missing 
data 

(e.g. assumptions made for 

ITT analysis) 


72. Assumed risk estimate 
(e.g. baseline or population 
risk noted in Background) 


73. Notes: 


9. Results 


Copy and paste the appropriate table for each outcome, including additional 


tables for each time point 


and subgroup as required. 


For randomised or non-randomised trial - Dichotomous outcome 


Description as stated in report/paper 


Location 
in text 
pey 
fig/table) 


74. Comparison 


75. Outcome 


76. Subgroup 
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Description as stated in report/paper 


Location 
in text 
(pg & T 

fig/table) 


77. Time point 
(specify whether 
from start or end of 
intervention) 


78. Results Intervention Comparison 
Note whether: 
... post-intervention No. No. No. No. 
OR events | participants | events | participants 
... change from baseline 
And whether 
... Adjusted OR 
... Unadjusted 
79. Baseline data Intervention Comparison 
No. No. No. No. 
events | participants | events | participants 


80. No. missing 
participants and 
reasons 


81. No. participants 
moved from 
other group and 
reasons 


82. Any other 
results reported 


83. Unit of analysis 
(e.g. by individuals, 
health professional, 
practice, hospital, 
community) 
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Description as stated in report/paper 


Location 
in text 
(pg & Y 

fig/table) 


84. Statistical 
methods 
used and 


of these 
methods 


correlation) 


appropriateness 


(e.g. adjustment for 


85. Reanalysis 
required? 

(if yes, specify why, 

e.g. correlation 

adjustment) 


Yes/ No/Unclear 


86. Reanalysis 
possible? 


Yes/ No/Unclear 


87. Reanalysed 
results 


88. Notes: 


For randomised or non-randomised trial - Continuous outcome 


Description as stated in report/paper 


Location 
in text 
(pe &Y/ 

fig/table) 


89. Comparison 


90. Outcome 


91. Subgroup 


92. Time point 


(specify whether from start 
or end of intervention) 


93. Post-intervention or 
change from baseline? 
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Description as stated in report/paper | Location 


in text 
(pp oY 
fig/table) 
94. Results Intervention Comparison 
Note whether: 
... post- Mean | SD (or | No. | Mean | SD (or | No. 
intervention OR other | partici- other par- 
... change from vari- pants vari- tici- 
baseline ance) ance) | pants 
And whether 
... Adjusted OR 
... Unadjusted 
95. Baseline Intervention Comparison 
data 
Mean | SD (or | No. Mean | SD (or | No. 
anoth- | partici- other | par- 
er vari-| pants vari- tici- 
ance) ance) | pants 


96. No. missing 
participants and 
reasons 


97. No. participants 
moved from other 
group and reasons 


98. Any other results 
reported 


99. Unit of analysis 
(e.g. by individuals, health 
professional, practice, 
hospital, community) 


100. Statistical 
methods used and 
appropriateness of 
these methods 

(e.g. adjustment for 

correlation) 


101. Reanalysis required? | ... 
(if yes, specify why) Yes/No/ 
Unclear 
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Description as stated in report/paper | Location 
in text 
(pe EY 
fig/table) 
102. Reanalysis possible? | ... 
Yes/No/ 
Unclear 
103. Reanalysed results 
104. Notes: 
For randomised or non-randomised trial - Another outcome 
Description as stated in report/paper Location 
in text 
(ps & Wig! 


table) 


105. Comparison 


106. Outcome 


107. Subgroup 


108. Time point 
(specify whether 
from start or end of 
intervention) 


109. Type of 
outcome 

110. Results Inter- SD (or Control | SD (or 

vention other result other 
result variance) vari- 
ance) 
Overall results SE (or other 
variance) 
111. No. participant | Intervention Control 
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Description as stated in report/paper 


Location 
in text 
(pg & Wfig! 
table) 


112. No. missing 
participants and 
reasons 


113. No. participants 
moved from 
other group and 
reasons 


114. Any other 
results reported 


115. Unit of analysis 
(e.g. by individuals, 
health professional, 
practice, hospital, 
community) 


116. Statistical 
methods 
used and 
appropriateness 
of these 
methods 


117. Reanalysis 
required? 
(if yes, specify why) 


118. Reanalysis 
possible? 


119. Unit of analysis 
(e.g. by individuals, 
health professional, 
practice, hospital, 
community) 


120. Notes 
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For controlled before-after study 


Description as stated in report/paper 


Location in 

text 

(pg & Ufig/ 
table) 


121. Comparison 


122. Outcome 


123. Subgroup 


124. Timepoint 
(specify whether 
from start or end of 
intervention) 


5. Post- 
intervention or 
change from 
baseline? 


126. Results 


Interven- SD (or 
tion re- other 
sult vari- 
ance) 


Control | SD (or 
result other 
vari- 
ance) 


Overall results 


SE (or other 
variance) 


127. No. 
participants 


Intervention 


Control 


128. No. missing 
participants 
and reasons 


129. No. 
participants 
moved from 
other group 
and reasons 


130. Any other 
results 
reported 
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Description as stated in report/paper 


Location in 

text 

(pg & Yfig/ 
table) 


131. Unit of 
analysis 

(individuals, cluster/ 

groups or body parts) 


132. Statistical 
methods used 
and appropria- 
teness of these 
methods 


133. Reanalysis 
required? 
(specify) 


Yes/No/ 
Unclear 


134. Reanalysis 
possible? 


135. Reanalysed 
results 


Yes/ No/ 
Unclear 


136. Notes: 


For interrupted time series or repeated measures study 


Description as stated in report/ 
paper 


Location in 
text 

(pg & Wig! 
table) 


137. Comparison 


138. Outcome 


139. Subgroup 


140. Length of 
timepoints 
measured 

(e.g. days, months) 


Total period measured 


48 


Steps in a systematic review 


Description as stated in report/ 


Location in 


paper text 
(pg & Tfis/ 
table) 
141. No. participants 
measured 
142. No. missing 
participants and 
reasons 
143. No. timepoints 8. Pre- 9. Post- 
measured intervention intervention 


144. Mean value 
(with variance measure) 


145. Difference in 
means (post — 
pre) 


146. Percent relative 
change 


147. Result reported 
by authors 
(with variance measure) 


148. Unit of analysis 
(individuals or cluster/ 
groups) 


149. Statistical 
methods 
used and 
appropriateness 
of these methods 


150. Reanalysis 
required? 
(specify) 


Yes/Nol 
Unclear 


151. Reanalysis 
possible? 


Yes/No/ 
Unclear 


152. Individual 
timepoint results 
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Description as stated in report/ 


Location in 


paper text 
(pg & Whig! 
table) 
153. Read from figure? | ... 
Yes/No/ 
Unclear 
154. Reanalysed Change in Change in 
results level SE slope SE 


155. Notes: 


10. Applicability 


156. Have important 
populations 
been excluded 
from the study? 

(consider disadvantaged 

populations, and 

possible differences in 
the intervention effect) 


Yes/ No/Unclear 


157. Is the 
intervention 
likely to be 
aimed at 
disadvantaged 
groups? 

(e.g. lower socioeconomic 

groups) 


Yes/ No/Unclear 


158. Does the study 
directly address 
the review 
question? 

(any issues of partial or 

indirect applicability) 


Yes/ No/Unclear 


159. Notes: 
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11. Other information 


160. Key conclusions 
of study authors 


Description as stated 
in report/paper 


Location in text 
(pg & Y/fig/table) 


161. References to 
other relevant 
studies 


162. Correspondence 
required for 
further study 
information 

(what and from whom) 


163. Further study 
information 
requested 

(from whom, what and 
when) 


164. Correspondence 
received 

(from whom, what and 
when) 


165. Notes: 


3.10 Quality assessment using critical appraisal tools 


The indicators for quality assessment of the included studies should be 
considered and can be integrated into the study coding form. Critical 
appraisal assesses quality and relevance of the research and is defined as 
‘the process of carefully and systematically examining research to judge its 
trustworthiness and relevance in a particular context”. Critical appraisal is 
a complex process and is guided by the research question, and associated 
review method. It is vital that the assessment process be standardized and 
transparent as possible (see Box 7). 
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Box7-Description of critical appraisal process 


A critical appraisal was conducted for the observational studies included in the 
Meta-Analysis, using the Critical Appraisal Skills Programme (UK) checklist, 
assessing the validity of the results from each study on a scale of high, medium, 
and satisfactory: high quality, the study was prospective and scored well on main 
quality parameters such as study method, result validity, precision of outcomes, 
and generalizability; medium quality, study method was sound and results were 
presented with precision; satisfactory quality, the study did not score well or did 
not contain any information on the main quality parameters such as study method, 
result validity, precision of outcomes, or generalizability. 


Adapted from Zhou, Shukla, John, & Chen (2015)?° 


A generic assessment looks at the quality of the execution of the study, 
however may not necessarily consider whether the study is a good fit for 
answering the review question”. Review-specific judgements assess the 
appropriateness of the study design and analysis for answering the research 
question or how well matched the study is to the focus of the review in 
terms of its topic. Depending on the type of review undertaken, reviews 
may consider both of these assessments or only one. 


The precise order in which critical appraisal and data extraction are 
undertaken varies from one systematic review to another depending on the 
type of systematic review. Mostly, there is an iterative relationship between 
the two, and there is no set guideline as to which should come first, however 
in interventional systematic reviews, risk of bias is assessed before data 
extraction for quantitative synthesis in Meta-Analysis 


There are various checklists of critical appraisal tools that one can use. 
However, the selection of the checklist should be explained (such as use 
for RCT or non-RCT study designs) or adapt them to their own review 
with the decisions stated and justified. Similar to the screening process, it 
is advisable that two reviewers independently conduct the critical appraisal 
of the selected studies and group the studies into high, medium and low 
quality. 
Table 8 - Critical appraisal of cohort studies example 


1. |Isthe research |1 1 1 1 1 1 
aim clearly 
stated? (Yes/ 
No) 
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Description of 
the context? 
(Yes/No) 


Description of 
the sampling 
procedures? 
(Yes/No) 


Are sample 
characteristics 
sufficiently 
reported? 
(sample size, 
location, and 
at least one 
additional 
characteristic?) 


(Yes/No) 


Is it clear how 
the data were 
collected (e.g. 
for interviews), 
is there an 
indication of 
how interviews 
were 
conducted? 
(Yes/No) 


Methods of 
recording of 
data reported? 
(Yes/No) 


Methods 

of analysis 
explicitly 
stated? (Yes/ 
No) 


Did the study 
address 

a clearly 
focussed issue? 


(Yes/No) 


Was the cohort 
recruited in 

an acceptable 
way? (Yes/No) 
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10. | Was the 1 al il il 1 0 
exposure 
accurately 
measured to 
minimise bias? 
(Yes/No) 


11. | Was the il 1 1 il 1 0 
outcome 
accurately 
measured to 
minimise bias? 
(Yes/No) 


12. | Have the il 1 il 1 il 1 
authors 
identified all 
important 
confounding 
factors? (Yes/ 
No) 


13. | Have they 1 1 1 1 1 1 
taken account 
of the 
confounding 
factors in the 
design and/or 
analysis? (Yes/ 
No) 


14. | Was the follow 
up of subjects 
complete 
enough? (Yes/ 
No) 


= 
Es 
= 
ren 
= 
E 


15. | Was the follow 
up of subjects 
long enough? 
(Yes/No) 


(Adapted from Panda, Dror, Koehlmoos, et al., 2013)7 
3.11 Quality of evidence 


GRADE (Grading of Recommendations, Assessment, Development 
and Evaluations) is a transparent framework for developing and 
presenting summaries of evidence. It is the most widely adopted tool 
for grading the quality of evidence and for making clinical/public health 
recommendations. 
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With regards to the systematic review question, the quality of evidence for 
each study outcome is rated using GRADE quality rating. GRADE has four 
levels of evidence- also known as certainty in evidence or quality of evidence: 
very low, low, moderate, and high (Table 9). Evidence from randomised 
controlled trials starts at a high quality and evidence from observational 
data starts at low quality due to the residual confounding. 


The certainty in evidence is increased or decreased for reasons as described 
below. 


Table 9: GRADE certainty ratings 
Certainty What it means 


Verylow The true effect is probably markedly different from the 
estimated effect 


Low The true effect might be markedly different from the estimated 
Effect 


Moderate The authors believe that the true effect is probably close to the 
estimated effect 


High The authors have a lot of confidence that the true effect is 
Similar to the estimated effect 


For each risk of bias, imprecision, inconsistency, indirectness, and publication 
bias, authors have the option of decreasing their level of certainty by one 
or two levels (e.g. from high to moderate), using a ‘Summary of findings’ 
table. 


Standard Cochrane ‘Summary of findings’ tables includes the following 
elements using one of the accepted formats. Further guidance on each of 
these are available on Cochrane Training website. 


1. A brief description of the comparison addressed in the ‘Summary 
of findings’ table, including both the experimental and comparison 
interventions. 


2. A list of the most critical and/or important health outcomes, both 
desirable and undesirable, limited to seven or fewer outcomes. 


3. A measure of the typical burden of each outcomes (e.g. illustrative risk, 
or illustrative mean, on comparator intervention). 


4. The absolute and relative magnitude of effect are measured for each (if 
both are appropriate). 
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5. The numbers of participants and studies contributing to the analysis of 
each outcomes. 


6. A GRADE assessment of the overall certainty of the body of evidence 
for each outcome (which may vary by outcome). 


7. Space for comments. 
8. Explanations (formerly known as footnotes). 


‘Summary of findings’ tables are supported by detailed tables, known as 
‘evidence profiles’, that provide greater detail than the tables of both of 
the individual considerations feeding into the grading of certainty and of 
the results of the studies. An example of a ‘Summary of findings’ table is 
provided in table 10. 


Table 10: Summary of findings 


Ciclopirox 8% lacquer compared to Vehicle for fungal infections of the toenails” 


Patient or population: people with fungal infections of the toenails 
Intervention: Ciclopirox 8% lacquer 

Comparison: Vehicle 

Setting: Outpatient clinics 


Complete Study population RR9.29 | 460(2 @@@BO NNTB=3 
"8 als (1.72 to | RCTs) L 
cure: .72 to s 
weeks = par ICW 50.14) ie 
Rees eee 41 per 1000 : 

Adverse (8 to 219) . RR 1.61 460 PARO The most 

events Study population (0.89 to 2 Low commonly 

Gey, operi 282)" RCT n 

treatment, E UU events were 
z (62 to 204) aie 

collected application 

over the site reactions 

course (transient 

of T a 

studies, urning, or 

they are not pain with 

measured/ treatment 

T a 

or specific mi 
timepoints) 
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erythema 

in the skin 
surrounding 
the nail), and 
alterations in 
nail colour 

or shape. 
These adverse 
reactions did 


not require 
additional 
treatment. 

Mycologi- Study population RR3.15 460 PADO NNIB=2 

cal cure: 48 96 per 1000 (1.93 to (2 Moderate? 

weeks 303 per 1000 5.12) | RCTs) 

(185 to 492) 

Clinical - - - - - - 

cure- not 

measured 


*The risk in the intervention group (and its 95% CI) is based on the assumed risk in the 
comparison group and the relative effect of the intervention (and its 95% CI). 


CI: Confidence interval; RR: Risk ratio; NNTB: Number needed to treat for an additional 
beneficial outcome 


GRADE Working Group grades of evidence 


High quality: We are very confident that the true effect lies close to that of the estimate 
of the effect 


Moderate quality: We are moderately confident in the effect estimate: The true effect 
is likely to be close to the estimate of the effect, but there is a possibility that it is 
substantiallydifferent 


Low quality: Our confidence in the effect estimate is limited: The true effect may be 
substantially different from the estimate of the effect 


Very low quality: We have very little confidence in the effect estimate: The true effect is 
likely to be substantially different from the estimate of effect 
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“Downgraded by two levels to low-quality evidence: one level due to imprecision as 
there are very large and variable confidence intervals across both studies, with low 
event rates (fewer than 100). Also downgraded by one level due to risk of bias, as most 
information is from studies at unclear risk of bias. 


Downgraded by two levels to low-quality evidence: one level due to imprecision since 
the 95% CI includes both a meaningful increase in risk, and no increase in risk, and one 
level due to risk of bias, as most information is from studies at unclear risk of bias. 


Adapted from: Topical and device-based treatments for fungal infections of 
the toenails”. 
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4. Meta-Analysis 


Meta-Analysis is a process of using quantitative methods to summarize 
the results from multiple studies, obtained and critically reviewed using 
a rigorous process (to minimize bias) for identifying, appraising, and 
synthesizing studies to answer a specific question and draw conclusions 
about the data gathered. The purpose of this process is to gain a summary 
statistic (i.e., a measure of a single effect) that represents the effect of the 
intervention across multiple studies. Gene V Glass coined the term “Meta- 
Analysis “in 1976. Since then the popularity of Meta-Analysis has increased 
significantly. The purpose is to increase the power of the evidence generated 
by combining small studies and improve the precision of the estimates by 
reducing uncertainty. It is similar to a simple cross-sectional study, in which 
the subjects are individual studies rather than individual persons. 


A systematic review is a meta-analytic review only if it includes a 
quantitative estimation of the magnitude of the effect and its uncertainty 
(confidence limits). Collection of studies that examine the same phenomenon 
or relationships can be combined by using meta-analytical procedures 
and empirical observations. Thus, Meta-Analysis can be used to combine 
outcomes of treatment and comparison groups in randomized controlled 
trials; prevalence rates in cross sectional studies; correlation coefficients in 
studies of association or pre-post /before-after event rates in before after 
studies. Meta-Analysis combines effect sizes from different studies to 
provide an overall estimate while examining and quantifying the variability 
among study findings and the effect of sampling on primary studies. 


The process of conducting a Meta-Analysis is similar to any other empirical 
study. It includes five steps: 


1. Formulating a research question (part of a systematic review) 
Defining a research question 
Define Inclusion/Exclusion criteria 
Locating and selecting studies 

2. Data collection and evaluation 
Data extraction 
Define effect estimates 
Tabulation of relevant parameters 

3. Data analysis and interpretation 
Testing homogeneity of studies 
Investigate sources of heterogeneity 
Choose appropriate statistical techniques 
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4. Presentation 


Based on the outcome of interest the data from primary studies are extracted 
from the results section. 


4.1 Effect measures 


Continuous outcome 


For each individual study, we assume two underlying populations 
representing the experimental versus control groups on a continuous 
outcome. Let Me and u, be the experimental and control population means, 
and gf and oe be the population variances, respectively. Such a design is 
applicable in studies that evaluate treatment outcome in behavioural sciences, 
education, medicine, etc. Under the assumptions of normal distribution 
and homoscedasticity, the usual parametric effect-size is the standardized 
mean difference (SMD), which is the difference between the experimental 
and control population means, u, and uy divided by the pooled population 
standard deviation. 


There are two approaches for estimating the SMD, one is the Cohens d 
method and the other is the Hedges’g method (with a small correction for 
small sample bias). 


Cohen’s d: Hedges’ g 


Cohen's d = == c(m)=1-3/(4m-1) with 
Spooled 


m=n, + nz — 2 


p _ |™- 1)s7 + (n2 — 1)s5 SE (g)= c(m)x SE(d) 
— ny + ng — 2 
d=c(m)XCohen's d) 
SE(d) = > + z + i 
Ng 2 (nı + fig = 2) 
Here 


nı=sample size of group 1; n2 =sample size of group 2 
xı = mean of group 1 
X> = mean of group 2 
s1= s.d of group 1 
s2 = s.d of group 2 
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Meta-analysts have provided arbitrary cut off points of 0.20, 0.50, and 0.80 to 
represent a small moderate and large effect. Sometimes SMD can be converted 
to odds ratios especially if there are studies which report results as continuous 
measurements and there are others which report the results as a binary 
outcome. In such cases the SMD can be converted to odds ratio by the formula 
m 
OR=—=SMD 
vy 


Correlation 


For studies that report correlation between continuous variables, the 
correlation coefficient r itself can serve as the effect size. The standard error of 
r is approximately 


SE(r) = = where n is the sample size. 
yn-1 


To obtain the 95% confidence interval for the correlation coefficient r it is first 
transformed into Z, where 
(i+T) 
in 


— G-r) 
a 


The log upper and lower bounds of Z,. are 
/Vn-3) 
The 95% confidence interval is= [(e7"-1) /(e7"+1), (e7"-1)/(e7+1)] 


L=z -= (Zu /Vn-3) U=z, +(z 


1-0/2 


If tests of association are combined, then the correlation coefficient(r) is the 
effect measure. 


Odds ratio, Risk Ratio or Risk Difference 


For data from a prospective study, such as a randomized trial, data is 
originally reported as the number of events and non-events in two groups 
in a 2X2 table: 


| Non events — Total 
Experimental intervention a b a+b 
Control intervention € d c+d 
Total atc b+d a+b+c+d 


Typically, the effect measures are the risk ratio, odds ratio, and/or risk 
difference. These are defined as follows: 
Risk of event in the experimental group _ a/(a+b) 


Risk Ratio=RR = 
Risk of event in the control group c/(c+d) 
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with the standard error of the log relative risk being 


1 1 1 1 
SE(In(RR)) = = ei ae ase 


And 95% confidence interval 
95% CI = exp(In(RR)-1.96x SE(In(RR)) to exp(In(RR)+1.96xSE(In(RR)) 


Odds of event in the experimental group a/b axd 
Odds Ratio= 0R:— TF SE 
Odds of event in the control group c/d bxc 


With the standard error of the log odds ratio being 
E oe T A 
SE(In(OR)) = £ + i + a + i 
And 95% confidence interval 
95% CI = exp(In (OR)-1.96x SE (In (OR)) to exp (In (OR)+1.96xSE(In (OR)) 


Risk difference=RD= risk of event in experimental group-risk of event in the control 
group=_“__ £ 


ath ctd 
axb cxd 


with the standard error being SE(RD) = casa e 


and 95% confidence interval 
95% CI = RD-1.96x SE (RD) to RD+1.96xSE (RD) 


Effect measures and their standard errors are extracted from primary studies 
by the reviewers in a tabular form for all the studies. 


4.2 Heterogeneity 


In Meta-Analysis, effect sizes from different studies are combined to obtain 
asummary effect. Heterogeneity in Meta-Analysis refers to the variation in 
the outcomes from different studies. Although results from multiple studies 
differ to some degree, heterogeneity occurs when their underlying target 
parameters differ. The heterogeneity in effect measures between studies 
could arise due to several factors: 


e Study design (inclusion criteria, treatment, duration) 
e Study quality (randomisation, blinding etc) 

e Individual level (prognostic factors) 

e Outcomes (chance results) 

e inadequate sample size 

e differences in participants 

e differences in doses of intervention 
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e differences in study design 

e differences in measuring instruments 
e different patient follow-up 

e different statistical analysis 

e different reporting 

e different patient response 


It is important to understand the factors responsible for heterogeneity since 
these could influence the overall outcome of the Meta-Analysis. 


The classical measure of statistical heterogeneity is Cochrane’s Q test which 
is calculated as the weighted sum of squared differences between individual 
study effects and the pooled effect across studies, with the weights being 
those used in the pooling method. Thus if Y4, Va, Yz, Va,-+++¥Vy are the 
effect estimates from k different studies and y is the summary effect then 
Q = Xw; (y;—y)* where w, is the study weighti.e. inverse of the study’s 
variance and the summation runs over all the k studies. Q is distributed as a 
chi-square statistic with k-1 degrees of freedom. The value of Q depends on 
the number of studies, how much each of the effect estimates deviate from 
the summary effect and the precision of the studies. If the standard error 
of an effect size is very low (and thus the precision is very high) even small 
deviations from the summary effect will be given a higher weight, leading 
to higher values of Q. 


Q has low power as a comprehensive test of heterogeneity” when the 
number of studies is small, i.e. in most meta-analyses. Conversely, Q has too 
much power as a test of heterogeneity if the number of studies is large®. Q 
forms part of the DerSimonian-Laird random effects pooling method*'. An 
additional test, due to Breslow and Day”, is provided with the odds ratio 
Meta-Analysis. 


It is arguably not possible to examine the null hypothesis that all studies 
are evaluating the same effect, by considering only the summary data from 
the studies: The heterogeneity test results should be considered alongside 
a qualitative assessment of the combinability of studies in a systematic 
review. 


To solve the problems of the Q statistic and the non-comparability of the 
between-studies variance, among meta-analyses with different effect-size 
metrics, Higgins and Thompson” have proposed the P index. The P index 
quantifies the extent of heterogeneity from a collection of effect sizes by 
comparing the Q value to its expected value assuming homogeneity, that is, 
to its degrees of freedom (df = k - 1): I? = 100% x (Q-df)/Q. It is an intuitive 
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and simple expression of the inconsistency of studies’ results. Unlike Q it 
does not inherently depend upon the number of studies considered. 


The P statistic describes the percentage of variation across studies that is 
due to heterogeneity rather than chance’. 


A rough guide to interpretation of P is as follows’: 
0-40%: might not be important 

30-60%: may be moderate heterogeneity 

50-90%: may represent substantial heterogeneity 
75-100% considerable heterogeneity 


4.3 Graphical methods for identification of heterogeneity 


A significant Q test indicates heterogeneity which is then tested further to 
identify the sources of heterogeneity. If heterogeneity is found then various 
methods could be adopted for investigating the possible reasons for it say for 
e.g. sensitivity or subgroup analysis, graphical methods, or Meta-regression. 
Subgroup analyses are done for subsets of participants either by gender or 
age group or by location or treatment regimen. One method of identifying 
heterogeneity is using graphs. There are many types of graphs that identify 
heterogeneity between the studies. Some of these are: 


4.3.1 L’ Abbé plot 


A plot showing the observed event rate in the experimental group plotted 
against the observed event rate in the control group for each study. L’ Abbé 
plots may be used to view the range of event rates among the trials, to 
highlight excessive heterogeneity. 


Figure 4: LAbbe Plot 


L'Abbe plot (symbol size represents sample size) 


Experimental percent 


0 20 40 60 80 100 
Control percent 
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4.3.2 Galbraith plot 


A plot of a standardized intervention effect (intervention effect divided 
by its standard error) against the reciprocal of the standard error 
(precision) (ES vs 1/SE). Imprecise estimates of effect lie near the origin, 
and precise estimates further away, giving the correct impression of 
being more informative. Vertical variation in points describes the extent 
of heterogeneity. Galbraith plots facilitate examination of heterogeneity, 
including detection of outliers. 


Figure 5: Galbraith Plot 


D) Galbraith Plot 
4 


Z Score 
bak bon 


-- —- + — 


a oe ae oe Oe ie e 
Inverse Standard Error 


4.3.3 Baujat plot 


This plot allows for identifying trials, groups of trials or groups of patients 
that are sources of heterogeneity and for quantifying the contribution of 
these trials to the overall result. Each trial is represented by a dot on a 2-D 
graph. The X axis represents the contribution of the trial to the overall 
Cochran Q-test for heterogeneity. The Y axis represents the influence of the 
trial, defined as the standardized squared difference between the treatment 
effects estimated with and without the trial. This method identifies trials 
that contribute considerably to the overall heterogeneity and have a strong 
influence on the overall result. 
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Figure 6: Baujat Plot 
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(Adapted from: Graphical displays for Meta-Analysis: An overview with 
suggestions for practice” ) 


4.3.4 Funnel plot 


The funnel plot is a scatter plot of effect estimate against a measure of 
precision (or study size). Mathematically, it plots the effect estimates of 
the studies against their standard errors. Funnel plots are used primarily 
as a visual aid for detecting bias or heterogeneity based on the level of 
asymmetry. A symmetric funnel shape plot indicates increasing scatter 
with decreasing precision (Figure 7). An asymmetric funnel may be the 
result of reporting bias (non-availability of results from small studies with 
non-significant results) to a systematic difference between smaller and 
larger studies, or to the presence of subsets of studies with different mean 
effect sizes Asymmetry can also arise with an inappropriate choice of 
effect measure, with an inappropriate choice of precision measure, with 
multiple inclusion of smaller (or larger) studies, or by chance. Whatever the 
cause, an asymmetric funnel plot leads to doubts over the appropriateness 
of a simple Meta-Analysis and suggests the need to investigate possible 
causes. 
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Figure 7: Funnel Plot 
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4.4 Meta-regression 


Whenheterogeneity isidentified betweenstudies, the sources ofheterogeneity 
need to be obtained. Metaregression like linear regression allows to study the 
effect of continuous, as well as categorical, characteristics of the studies on 
the effect size. In Meta-regression, the outcome variable is the effect estimate 
(for example, a standardised mean difference, a risk difference, a log odds 
ratio or a log risk ratio). The explanatory variables are characteristics of the 
studies that might influence the size of the intervention effect. The regression 
coefficient obtained from a Meta-regression analysis will describe how the 
outcome variable (the intervention effect) changes with a unit increase 
in the explanatory variable (the potential effect modifier). The statistical 
significance of the regression coefficient is a test of whether there is a linear 
relationship between the intervention effect and the explanatory variable. 
Meta-regression should generally not be considered when there are fewer 
than ten studies in a Meta-Analysis. Meta-regression may be performed 
using the ‘metareg’ macro available for the Stata statistical package. 


The next step is to analyze the data using either of the two approaches the 
Fixed effect method and the Random effect method. 


The Fixed effects model is based on the assumption that a single common 
(or ‘fixed’) effect underlies every study in the Meta-Analysis. This approach 
is usually applied when P is small. It tries to answer the question that “Did 
the treatment produce any benefit on an average in the studies at hand?” 
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The basic assumptions are: 


° Studies use identical methods, patients, and measurements; 
° Produce identical results; 
e Differences are only due to within-study variation. 


In this model, all of the observed differences between studies is due to 
chance. 


The Random effects model is based on the assumption that individual 
studies estimate different true effects. These true effects have a distribution 
with some central value and some degree of variability. Thus, the random 
effect model allows for random error plus between the studies variability, 
results in wider confidence intervals and tends to give larger weights to 
smaller studies than fixed effects model. 


The basic assumptions are: 

e Studies are a random sample from the universe of all possible studies 

e Differences occur both due to between-study and due to within-study 
variability 


4.5 Presentation 


In Meta-Analysis the results are presented in a graphical form. A forest plot 
is a graphical representation that summarises all the essential information of 
a Meta-Analysis. Often there are 6 columns in a forest plot (Figure 8). 


Figure 8: Forest plot comparing two interventions 


Column 1 Column 2 Column 3 Column 4 Column 5 Column 6 
risperidone clozapine RR Weight RR 
Study nM nm (95%CI Fixed) % (95%CI Fixed) 
Clozapine 1996 22/39 6/20 13.2 1 .88[0.91 3.88) 
Clozapine 1998a 9/43 9/43 15.0 1,00(0.44 2.27) 
Clozapine 1998b 34/135 33/138 627 0.91[0.61 1.36) 
x Clozapine 1999 0/15 0414 00 Not Estimable 
Clozapine 2000 1/9 6/11 3.0 0.20[0.03,1 .40] 
Total(95%Cl) 66/241 59 4226 100.0 0.99[0.73,1 35] 


Test for heterogeneity chi-square=5.76 df=3 p=0.12 
Test for overall effect z=-0.06 p=1 


1 2 1 5 10 
Favours risperidone Favours clozapine 


Column 1: Studies IDs 


The first column shows the list of included studies. These are depicted as 
name of the first author and the year of publication 


Column 2 and column 3: Experimental group n/N and Control group n/N 


These columns indicate the number of patients having the outcome of 
interest(n) out of the total number of patients (N) in the experimental or 


control group. 
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Column 4: Relative risk (fixed) 95% CI 


This displays the study results. The boxes show the effect estimates from the 
single studies, while the diamond shows the pooled result. The horizontal 
lines through the boxes(whiskers) illustrate the length of the confidence 
interval. Longer whiskers mean, a wider confidence interval. For the 
summary measure, the confidence interval is the width of the diamond. The 
vertical line is the line of no effect which could be at 1 for depicting ORs or 
RRs and 0 when depicting continuous measurements. 


If the boxes with whiskers and diamond lie away from the vertical line then 
the outcome is statistically significant. If the outcome of interest is adverse 
(e.g. mortality), the results to the left of the vertical line favour the intervention 
over the control. That is, if result estimates are located to the left, it means 
that the outcome of interest (e.g. mortality) occurred less frequently in the 
intervention group than in the control group (ratio < 1). 


If the outcome of interest is desirable (e.g. remission), the results to the right 
of the vertical line favour the intervention over the control. That is, if the 
result estimates are located to the right, it means that the outcome of interest 
(e.g. remission) occurred more frequently in the intervention group than in 
the control group (ratio > 1). 


The last possibility: if the diamond touches the vertical line, the overall 
(combined) result is not statistically significant. It means that the overall 
outcome rate in the intervention group is much the same as in the control 
group. This is the case in the figure above. 


Column 5: Weight (%) 


For the next column over, the weight (in %) indicates the influence an 
individual study has on the pooled result. In general, the bigger the sample 
size and the narrower the confidence interval (CI), the higher the percentage 
weight, the larger the box, and more the influence the study has on the 
pooled result. In general, the weights of effect sizes for individual studies is 
the inverse of variance. 


Column 6: Relative risk (fixed) 95% CI 


The rightmost column contains exactly the same information as is contained 
in the diagram in column 4, just in numerical format. So, we can observe the 
data both in picture and in numbers. This can be either the 95% CI of odds 
ratio (OR) or the 95% CI of relative risk (RR). The diagram above shows 
relative risk. When the 95% CI does not include 1, we can say the result is 
statistically significant. 
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More information is found in the lower left corner of the plot. 


The p-value indicates the level of statistical significance. If the diamond 
shape does not touch the line of no effect, the difference found between the 
two groups was statistically significant. In that case, the p-value is usually 
< 0.05. 


4.6Reporting a systematic review and Meta-Analysis 


The last step is the interpretation of the results, discussion of issues such 
as clinical applicability and writing of the manuscript for publication. 
To ensure that a systematic review is useful to the users, a transparent, 
complete and accurate account of why, what and how the review was done 
must be reported. Reviewers need to discuss the limitations of the primary 
studies included in their review, and limitations in how the review itself 
was conducted. Limitations of the primary studies, for example, may 
include issues relating to design flaws. Limitations of the review itself may 
include issues such as inclusion of only English language studies or inability 
to accurately interpret the summary estimates due to heterogeneity. A 
discussion of these limitations will enable readers to judge the strength of 
the evidence presented in the review. The review usually concludes with 
a discussion on the implications for clinical practice, and need for further 
research. If the evidence is strong and unequivocal, reviewers might 
recommend no further trials on that clinical question. Some reviews (e.g. 
reviews on screening tests such as mammography) may have important 
public health or policy implications that merit discussion. 


For writing the manuscript for publication, reviewers have two useful 
guides: the PRISMA 2020™ (Figure 9) guidelines for systematic reviews 
of studies that evaluate social, educational or health interventions and the 
MOOSE guidelines® for systematic reviews of observational studies. Many 
items of PRISMA 2020 are applicable to systematic reviews of studies on 
aetiology, prevalence or prognosis. Many journals now encourage authors 
to submit manuscripts formatted according to these guidelines. Moreover, 
these guidelines can serve as practical tools for the critical reader in assessing 
the quality of an individual Meta-Analysis. In addition to these guidelines, 
reviewers can find a variety of outstanding resources for conducting reviews 
on the list of websites mentioned in the previous chapter. 


The following is a list of expected information for a systematic review: 


1. A report of literature scoping containing combinations of search strings 
and the outcome of searches of different databases (this is usually as an 
appendix with the protocol). 
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2. A list of articles excluded after reading the full text, including reasons 
for exclusion 


(Note: a list of articles included is expected in the main text). 


3. A list of articles that could not be obtained at full text despite efforts 
such as contacting the corresponding author, help from librarians, 
managing editor of relevant Cochrane review groups etc.: such articles 
are therefore potentially relevant but not fully screened. 


4. Data extraction and quality assessment tables; for example, Excel files 
with data extracted from each included study (this may be included in 
the main text if a small number of studies is included). 


Example of Systematic Review Report: 


Panda, P, Dror, IH, Koehlmoos, TP, Hossain, SAS, John, D, Khan, JAM 
and Dror, DM (2016). What factors affect uptake of voluntary and community- 
based health insurance schemes in low-and middle-income countries? A systematic 
review, 3ie Systematic Review 27. London: International Initiative for Impact 
Evaluation (3ie) 


Figure 9: PRISMA 2020 Flow diagram 
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5. Systematic review and Meta-Analysis 
software 


5.1 Systematic review software 


A systematic review as we have discussed in the previous chapters follows 
some basic steps. There are some statistical software that help the researcher 
to conduct different steps of a systematic review. These software provide 
solutions for screening, searching and data analysis. some of the software 
that are useful during the systematic review are: 


Rayyan QCRI 


Rayyan is a freely available web and mobile application that helps expedite 
the initial screening of abstracts and titles. It is easy-to-handle, user- 
friendly and reduces the load of reviewers by performing quick tasks. The 
Rayyan software helps accelerate the initial screening of abstracts and titles 
using a process of semi-automation and data extraction with cloud-based 
architecture that allows it to scale accordingly during the peak times and as 
the number of users grows and they can create more reviews and upload 
more citations. Rayyan mobile app works in a way that one can download 
the review while online and then screen even in the absence of a network, 
and then, after getting the network, it will automatically sync back to server. 
The advantage of using Rayyan is that it learns from reviewers’ decision, 
that is, which study the reviewer wants to include and which to exclude. 
By removing stop words and stemming, Rayyan extracts all the possible 
combination of words and these are then used as features by a Support 
Vector Machine (SVM) classifier. SVM plays an important role as it learns 
from the included and excluded citations builds a model and a classifier 
accordingly. One can get access to Rayyan web by using the following 
link: https://rayyan-prod.qcri.org/welcome. Rayyan is integrated with Review 
Manager (RevMan), a Cochrane software used for preparing and maintaining 
Cochrane reviews. 


COVI DENCE 


One of the software used for systematic review management is COVIDENCE 
which is paid but free to use for those authoring Cochrane reviews. It is 
one of the most popular systematic review tools that can facilitate screening 
of abstracts and full text, data extraction, import citations from reference 
managers like EndNote. It is a core component of Cochrane’s review 
production toolkit. https://utas.libguides.com/SystematicReviews/Tools . 
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EPPI Reviewer 


EPPI (Evidence for Policy and Practice Information) Reviewer is free for 
all the Cochrane contributors. EPPI Reviewer has many features including 
reference management, data extraction, storage, annotation and coding of 
files, easy export of review data so that it can be used with other software 
applications. It can customize tools according to the demands of a specific 
review. You can create your account on EPPI Reviewer by clicking on the 
following link: 


https://eppi.ioe.ac.uk/EPPIReviewer-Web/home 


https://community.cochrane.org/help/tools-and-software/eppi-reviewer 
Abstrackr, SRDR Plus and OpenMeta 


Brown School of Public Health introduced softwares like Abstrackr, SRDR 
Plus and OpenMeta for systematic review and Meta-Analysis related work. 
Abstrackr software is used for screening of systematic reviews, citation and 
to upload your abstracts. SRDR (The Systematic Review Data Repository) 
Plus is a tool for extracting, managing and archiving data whereas OpenMeta 
is an open source platform to implement Meta-Analysis. One can create 
account on these three through this link: 


https://www.brown.edu/public-health/cesh/resources/software 


( https://utas.libguides.com/SystematicReviews/Tools ) 
DistillerSR 


DistillerSR is a priced software that automates the management of literature 
review using AI and intelligent workflows. It is integrated with EBSCO, 
Ovid and PubMed and other existing libraries and automatically updates the 
review when new published references on the topic of interest is added with 
its additional Module DistillerSR LitConnect. It reduces the review screening 
time by automatically categorizing the references. It helps detect and 
remove duplicates faster through AI powered screening and automatically 
identifies conflicts and disagreements between reviewers. The software has 
open access integrations thereby uploading full text copyright compliant 
documents from PMC. The DistillerSR CuratorCR module reduces data 
extraction time by preventing duplication of efforts. It is able to capture 
complex data and reduce the time of the reviewers in data cleaning and 
effect measure computations. It helps in building reports, PRISMA diagrams 
and other standard reports. Further details can be obtained at https://www. 
evidencepartners.com/. 
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CADI MA 


CADIMA is a free web-based tool that facilitates conduct of systematic 
review. The software allows unlimited multiple reviewers at any stage of 
the review process. Review questions in any discipline can be addressed 
in CADIMA. It allows automated duplicate removal, assures automated 
screening and allows bulk pdf upload. There are tutorials on the website for 
reviewers and the users can register for online workshops for understanding 
the functionality of the software. One can get more details from https://www. 
cadima.info/. 


SYSREV 


Sysrev is a web-based platform for data curation and systematic reviews. It 
provides an easy to access free platform in the public domain for collaborative 
systematic reviews. It allows users to upload documents, recruit reviewers, 
perform reviews and automate review tasks. Sysrev uses the FAIR- 
Findability, Accessibility, Interoperability, and Reuse of digital assets design 
that encourages better data stewardship and management to maximise data 
transparency and reproducibility. These principles incorporate assignment 
of unique identifiers for retrieval of metadata that supports reuse of data. It 
is a software that extracts data from various sources with a variety of import 
structures like XML, RIS citation format, PDF, JSON and HTML formats 
using a programming interface. It integrates data from pubmed.gov and 
clinicaltrials.gov. It generalizes the review process using a digital document. 
The user can get more details from https://sysrev.com/. 


There are several organizations that work on syntheses of evidence in the 
form of systematic reviews such as Equator Network, JBI Collaboration, 
Cochrane Collaboration and Campbell Collaboration. The Cochrane 
Collaboration helps people make informed decisions in healthcare by 
preparing, maintaining and promoting the accessibility of systematic reviews 
of healthcare interventions. The Campbell Collaboration is an international 
social science research network and both these use standards and guidelines 
for conducting and reporting high quality evidence. 


One can sign-up for Campbell network from the link given below: 
https://www.campbellcollaboration.org/ 


(https://campbellcollaboration.org/news-and-events/events/campbell-events/ 
leveraging-global-regional-and-local-evidence-for-evidence-based-policy- 
making-in-india.html) 
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The CDSR (Cochrane Database Systematic Review) consists of all systematic 
reviews prepared by Cochrane Review Group. Cochrane reviews are 
updated regularly as new studies occur because studies can change the 
conclusion of review. To sign-in for Cochrane library, one can get the access 
from the given link: https://www.cochranelibrary.com/ 


(https://www.cochranelibrary.com/cdsr/about-cdsr) 


The JBI (Joanna Briggs) Collaboration is one of the biggest collaborations 
that integrates evidence-based outcomes of healthcare globally which helps 
in decision-making. You can search for systematic review title by clicking on 
the following link: https://joannabriggs.org/ebp/systematic_review_register 


Also, the systematic review title registration form is available on the above 
link. 


JBI SUMARI is a software developed by JBI to implement the Meta-Analysis. 
If one wants to learn how JBI works in synthesis and Meta-Analysis, you can 
visit the following link: https://www.jbisumari.org/#tutorials 


(https://wiki.joannabriggs.org/display/JBCI/JBI+EBP+Directory) 


The EQUATOR wizard is very simple to use and easily understandable. 
The equator wizard includes common guidelines for generic study types. It 
has drop down boxes for selecting the area of interest. The EQUATOR also 
provides a book entitled Guidelines for Reporting Health Research that 
provides standardized reporting methods of important details in health 
research publications. 


https://www.equator-network.org/reporting-guidelines-study-design/ 
systematic-reviews-and-meta-analyses/? post_type=eq_guidelines&rgo=post_ 
date 


5.2 Meta-Analysis software 


Meta-Analysis can be done using various software and every software has its 
own pros and cons. Before performing Meta-Analysis on any software, it is 
essential to know the methodology adopted. RevMan, Metawin, Comprehensive 
Meta-Analysis (CMA), Microsoft Excel, R, Stata, SAS, NetMetaXL, Python etc. are 
some of the software for performing Meta-Analysis. Some of them are freely 
available such as R, Python and some software requires external libraries (add- 
ins). 


MS Excel 


One way to perform Meta-Analysis in MS Excel is by entering the formulae 
for the computation This requires complete knowledge of methods in Meta- 
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Analysis. One can make forest plot in Excel after doing step-by-step analysis 
such as calculating effect size, standard error, study weights, weighted effect 
size, Q and I? etc. It helps in performing fixed-effect and random-effect Meta- 
Analysis. One can draw a forest plot in MS Excel by using scatter plot with the 
error bars. 


The forest plot in MS Excel can not be customised to the need of the user. 
MS Excel is not freely available as it is a part of MS Office**. The Confidence 
interval calculator in MS Excel can be used to calculate confidence intervals for 
means or difference between means, proportions or odds, odds ratio, relative 
risk, sensitivity and specificity. 


Meta-Essentials is a tool designed for MS Excel that contains several 
spreadsheets to conduct Meta-Analysis by computing required statistics, tables 
and graphs automatically based upon the input given by the user. It consists 
of moderator analysis, sensitivity analysis, Egger’s regression, Galbraith plot, 
funnel plot, forest plot, Begg and Mazumdar rank correlation test etc. The 
spreadsheets are compatible with the MS Excel 2010, 2013 and 2016. Some 
worksheets of the workbook may work well with current versions but the 
earlier versions may not support the formatting and the formulae. One can 
download the user-manual for Meta-Essential tool from the link given below: 


https://www.erim.eurnl/research-facilities/meta-essentials/user-manual/ 


MetaEasy. MetaEasy is an add-in in MS-Excel that facilitates in performing 
Meta-Analysis. It is useful to produce reports, calculate the effect size, standard 
error etc. The add-in also supports advance Meta-Analysis such as maximum 
likelihood, profile likelihood and permutation methods. Overall, there are seven 
frequentist Meta-Analysis methods included in this add-in. In general, the 
forest plot contains a square for each study but MetaEasy also provides several 
squares in the forest plot for each study according to the multiple outcomes 
used in each study. It also calculates various heterogeneity measures such as 
Q-statistic, I? statistic and H, where H¥, is independent of the number of 
studies and ranges from 0 to œ. 0 indicates perfect homogeneity. This add-in 
is freely available on the website: http://www.statanalysis.co.uk/. MetaEasy 
consists of five worksheets in which the first the sheet is used for entering the 
data and it will automatically calculate the effect size and standard error”. 


Another add-in for Meta-Analysis in MS Excel is MetaXL that supports Quality 
Effects (QE) model and also inverse variance heterogeneity. MetaXL has 
some different features for detecting publication bias, which are, LFK index 
(quantitative measure for publication bias) and Doi plot instead of Funnel plot. 
With the release of advance version of MetaXL, it introduces Network Meta- 
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Analysis as well as cumulative Meta-Analysis. http://www.epigear.com/index_ 
files/metaxl.html 


Another MS Excel add-in is MIX 2.0 which is specially developed for educational 
purpose and can be freely downloaded from the link http://www.mix-for-Meta- 
Analysis.info. This platform also supports causal meta-analyses and the results 
obtained from MIX has been validated with Stata and Comprehensive Meta- 
Analysis version 2. 


NodeXL and NetMetaXL are other MS excel add-ins that supports network 
Meta-Analysis. 


R software 


R is an open-source software and has many packages to perform different 
types of meta-analyses such as standard inverse variance Meta-Analysis, three- 
level Meta-Analysis, network Meta-Analysis etc. One needs knowledge of R 
programming to use these packages. 


Metafor is the most commonly used package for Meta-Analysis in R. This 
package helps in calculating all types of effect sizes and also does data synthesis 
using fixed, random and mixed effect models. Metafor supports moderator 
analysis, Meta-regression and graphical plots like forest, Baujaut and funnel 
plot. This package also provides a variety of generalized linear models and 
meta-analytic multivariate /multilevel models. 


The package Meta provides trim-and-fill method to evaluate bias, fixed and 
random effect Meta-Analysis, cumulative Meta-Analysis, meta regression, 
leave-one-out Meta-Analysis, subgroup Meta-Analysis and various graphical 
plots such as funnel, forest, Galbraith, baujaut, LAbbe and bubble plot. 


The package robumeta is used to conduct robust variance estimation (RVE) 
Meta-regression using both large and small sample RVE estimators using 
various weighting schemes. These methods are distribution free and provide 
valid point estimates, standard errors and hypothesis tests even when the 
degree and structure of dependence between effect sizes are unknown. It also 
includes functions to perform sensitivity analyses under correlated effects 
weighting and producing RVE-based forest plots. 


The other packages such as DTA MA (Diagnostic Test Accuracy Meta- 
Analysis), dmeta and Meta-CART are also available to implement Meta- 
Analysis in R. A package metaDigitise helps in extracting descriptive statistics 
and raw data hidden in the figures (bar plot with standard error, box plot, 
scatter plot and also histogram) in primary research papers. 
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Python 


The package in python to execute Meta-Analysis is PythonMeta which has 
many features to combine effect sizes such as RR, RD, OR, MD or SMD. 
This package also helps the users to draw funnel plot and forest plots and to 
check heterogeneity viz. Q-statistic/chi-square test. The latest version of this 
package is Version1.11 (July 2019). https://pypi.org/project/PythonMeta/. 


The tool Pymeta is an online tool created and supported by PythonMeta 
package of Python Programming language. With the help of this tool, 
one can combine various effect measures and perform subgroup analysis, 
cumulative Meta-Analysis, sensitivity analysis and also has the power to 
draw plots like forest plot, funnel plot, cross-block plot etc. 


Stata 


Stata also has the feature to perform meta-analyses. There is a main 
community-contributed package metan that has all the commands required 
to perform standard Meta-Analysis. This package consists of several 
commands such as labbe, metareg, metafunnel, metap, metabias, metatrim, 
metaan etc. These commands are useful in the creation of graphical plots 
like Labbe plot, forest plot, funnel plot and effect size, Meta-regression etc. 
https://www.stata.com/support/faqs/statistics/Meta-Analysis/ 


The command meta supports effect size as well as summary statistics, forest 
plot, publication bias etc. The guide to perform Meta-Analysis on Stata can 
be downloaded from the following link: https://www.stata.com/manuals/ 
meta.pdf 
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6. Disseminating Systematic Review 
Findings 


Over the last decade, there have been substantial investments in the 
commissioning and funding of systematic reviews assessing effects of a 
range of different healthcare interventions. In order to improve the quality 
of healthcare, and health outcomes, the findings from systematic reviews 
need to be effectively communicated to practitioners and policy-makers. 
The following are some of the dissemination products for communicating 
findings from systematic reviews: 


6.1 Evidence Summary 


An Evidence Summary is a short one- or two-page document that describes 
in a lay and friendly language the findings from the best and most relevant 
evidence from systematic reviews on a particular health intervention with 
implications for further research. An Evidence Summary extracts information 
from systematic review(s), evaluates the information and presents the 
findings in a user-friendly manner such that decision-makers can quickly 
review the evidence and decide whether a particular innovation is likely to 
be effective in their own context. 


Structure and Content of an Evidence Summary 


1. Title 


The title usually consists of the topic reviewed and presented in the summary. 
It usually states the primary research question or issue of interest addressed 
in the summary. 


2. Key messages 


This section summarizes the research findings and outlines the key messages 
that one is trying to communicate. 


3. Background to the review question 


Under this section, one provides a brief background information on the 
topic being addressed by the evidence summary 


4. Methods (a summary of reviewed studies and sources of information) 


This section presents a summary of the reviewed studies and the respective 
sources of evidence that were used to draw the key messages and conclusions. 
It is important to highlight how the reviewed studies were searched and 
selected as a reliable source of evidence. 
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5. Evidence 


This section provides answers to the review question. It provides the level 
and quantity of evidence found for the review question. For clarity and 
impact, the evidence should be summarized in bullet points. 


6. Case studies 


It is usually helpful for practitioners and policy makers to also present 
studies that provide additional evidences relating to the review question if 
available. Case studies make an Evidence summary lively. 


References to resources containing information about these case studies 
should be given. 


7. References 


All references cited in the text should be included under this section. The 
use of a standard referencing style is highly recommended. 


8. Acknowledgements 


It is important to acknowledge all those that contribute to the process of 
putting together the evidence summary. 


1. Conflicts of interests must also be declared 
2. Additional Information 


Provide contact details including e-mail and phone numbers for readers to 
ascertain more information. 


For more details on Evidence Summary, please look at The Supporting Policy- 
Relevant Reviews and Trials (SUPPORT) project at www.supportsummaries. 


org 
6.2 Plain Language Summary (PLS) 


A Plain Language Summary (PLS) is a short, easy-to-read summary of 
a published systematic review. The main aim of the PLS is to provide 
information to a patient, carer, or practitioner on the key points of scientific 
evidence from a systematic review without getting into the clinical and 
synthesis details. A PLS is supposed to be clear, understandable, and 
accessible (i.e. open-source), especially for lay persons in the particular field 
of healthcare. Cochrane has published Standards for the reporting of Plain 
Language Summaries in new Cochrane Intervention Reviews (PLEACS). 
An example of PLS is given in box 8. 


80 


Disseminating Systematic Review Findings 


Box 8: Plain language summary example 


Nutritional labeling for healthier food or non-alcoholic drink purchasing and 
consumption 


Nutritional labelling to promote healthier consumption and purchasing of food 
or drinks 


A poor diet including excessive energy intake is an important cause of ill health. 
Nutritional labelling may help people to make healthier food choices. 


What is the aim of this review? 


This review investigated whether nutritional labels (i.e. labels providing 
information about nutritional content) persuade people to buy or consume 
different (healthy) kinds of food. We searched for all available evidence to answer 
this question and found 28 studies. 


Key messages 


There is evidence to suggest that nutritional labelling, with energy information 
(e.g. calorie counts) on menus, may reduce energy purchased in restaurants, but 
more high-quality studies are needed to make this finding more certain. 


What was studied in the review? 


Some studies assessed buying food or drinks from vending machines, grocery 
stores, restaurants, cafeterias, or coffee shops. Others assessed the amount of 
food or drink consumed during a snack or meal in an artificial setting or scenario 
(referred to as laboratory studies or settings). 


What are the main results of the review? 


Nutritional labelling on restaurant menus reduced the amount of energy (i.e. 
calories) purchased, but the quality of the three studies that contributed to this 
finding was low, so our confidence in the effect estimate is limited and may 
change with further studies. Eight studies assessed this same type of intervention 
in laboratory settings, but instead of evaluating how much energy participants 
purchased, these studies evaluated how much energy participants consumed. 
These studies did not conclusively demonstrate a reduction in energy consumed 
when menus or foods were labelled, and they were also of low quality. 


In addition, six laboratory studies assessed how much energy participants 
consumed when they were given one food or drink option with or without labels, 
and five laboratory studies assessed how much energy participants consumed 
when foods were experimentally labelled as low energy or low fat when they were 
actually high-energy foods (i.e. mislabelling). Results from these two groups of 
studies were inconclusive and of low, or in the case of mislabelling studies, very 
low quality. We found some studies that assessed labelling on vending machines 
and grocery stores, but their results were not easy to interpret, so we could not use 
them to inform this review. 


How up-to-date is this review? 


The evidence is current to 26 April 2017. 
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6.3 Policy briefs 


Reports that address the interests and needs of policy makers are referred to 
as policy briefs”. Policy briefs are based on systematic reviews to advance 
policymaking based on the best available evidence. The purpose of a policy 
brief is to create a short document providing findings and recommendations 
to an audience who may not necessarily be experts in that area. The audience 
for a policy brief can be the general public or particular entities of interest 
that seek solutions to problems or needs or who may require to be convinced 
of a different way of looking at an area of interest”. 


Examples of policy brief: 


1. Community-based health insurance: how to promote effective and 
equitable coverage? 


This brief is based on Factors affecting uptake of voluntary and community- 
based health insurance schemes in low- and middle income countries, 
3ie Systematic Review 27, by Pradeep Panda, Iddo H Dror, Tracey Perez 
Koehlmoos, SA Shahed Hossain, Denny John, Jahangir AM Khan and 
David M Dror. It synthesises evidence from 54 studies (36 quantitative, 12 
qualitative, and 6 mixed-method studies), covering 20 countries, mainly in 
Africa, South Asia and South East Asia, to understand the factors affecting 
uptake of CBHI schemes. Most studies reported on schemes in rural settings 
and in low-income countries, with only few lower-middle income countries 
and only very few upper-middle income countries(Adapted from the 
published policy brief).”” 


2. Oral Cholera Vaccines— worth a shot? 


This policy briefis based onasynthesis thathighlights that contrary to popular 
belief, cholera has its foothold firmly rooted in India. Frequently encountered 
symptoms of the disease are vomiting and at times, mild fever along with 
diarrhoea and faecal loss of salt and water leading to dehydration. Every year 
several outbreaks of the disease are reported from across the country. A few 
districts, based on the data from ‘Integrated Disease Surveillance Program’ 
(IDSP) (2011-2015), can even be labelled as endemic for cholera. However, 
we underline that the burden of cholera estimated for India is hamstrung 
by lack of robust surveillance and scarcity of incidence studies. Analysis of 
outbreak reports indicates that some settlements and populations are more 
vulnerable to diarrhoea and cholera compared to others. This heterogeneity 
provides an opportunity to prioritize areas for intervention but also cautions 
against applying incidence data obtained from one study from Kolkata to 
all of India. The short lasting nature of most of the recent cholera out-breaks 
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in India (2-3 weeks), requirement of two doses of the licensed OCV to be 
administered with a gap of 2 weeks in between and emerging evidence that 
considerable efficacy could be attained by single dose administration of the 
same vaccine call for pragmatic approaches. We recommend that a) existing 
surveillance system for cholera be strengthened, b) mapping of vulnerability 
to cholera as pertinent to population groups and geographical locations be 
continued and existing information be used for decision making, c) single 
dose bivalent killed OCV (licensed in India) be deployed in operational 
exploration mode in selected settings and in pre-cholera season and d) 
investment for safe water and sanitation as well as hygienic practices be 
boosted. The cost of inaction today could mean lives claimed by cholera in 
underserved areas, deepening of poverty and inequity, and perpetuation of 
expenses needed to tackle recurrent outbreaks of cholera and other diarrheal 
diseases tomorrow(Adapted from the published policy brief)”. 


3. Task shifting to optimise the roles of health workers to improve the 
delivery of maternal and child healthcare 


This policy brief was prepared by the Uganda country node of the Regional 
East African Community Health (REACH) policy initiative. The purpose was 
toinform deliberations among policy makers and stakeholders. It summarises 
the best available evidence regarding the design and implementation of 
policies for extending the use of non-medically trained primary health care 
workers to deliver cost effective maternal and child health interventions“. 
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